[qingyun-wu]

What changed, and why

On a terminal-run core, the legacy unsuffixed ag2.space bridge is usually the primary and is running. So "add s.space, then make it primary" fails today:

• set_primary is refused with legacy_primary_running while that bridge holds its lease.
• disconnect refuses the legacy row with legacy_row_unmanaged.

The only way through was to stop everything by hand, promote, and restart.

This PR adds one engine operation, promote(instance), that moves the primary while the old primary's bridge is still running:

1. Under the record lock it refuses a disabled row, notes the current owner of unaddressed messages (the primary row, or the legacy row while its bridge runs), clears the row's parked flag and makes it the only primary first, so launch_env gives its bridge REMOTE_PROACTIVE_ROOM.
2. If the row's bridge already runs (launched as a secondary, without the room) it is stopped; then the bridge is started. A failure raises new_primary_not_started.
3. It waits up to CONNECT_TIMEOUT_S (60 s, past the bridge's first long poll) for the new bridge's own status file, written by that pid, to show connected: true, meaning the gateway accepted its token. A 401, an exit or the deadline raises new_primary_not_connected. Steps 2 and 3 roll back on failure (primary and parked flags restored, the started bridge stopped, a stopped secondary started again); the old bridge is never touched.
4. Under one hold of the record lock it stops the old owner's bridge with the existing stop path (lease-proven pid, TERM, then wait for the lease to be released and the process to exit) and marks that row parked. A legacy bridge with no row of its own (a named row covers its server, so import-env gave it none) is stopped too and recorded as a record-level legacy_parked. First it checks that the row is still the primary and that the row to park is still there and unparked. If another writer changed either during the wait, it raises promote_conflict (409): the bridge this call started is stopped, the row's parked flag is restored, and the other writer's primary stands.

promote(''), back to the legacy row, needs that bridge running and runs step 4 only. That bridge settled the proactive room when it started, so when a named row was primary the answer carries relaunch_needed: true (restart it with bash src/restart.sh).

Ownership stays single throughout. Delivery is claim-based, and since #5960 every bridge reads owns_unaddressed from the record on each pass. So before the flip only the old bridge owns unaddressed messages and cards, after it only the new one does, and until the new bridge polls nobody does. Messages wait during that window; none are lost.

A refused stop (bridge_still_running or bridge_lease_indeterminate) rolls back the same way: the primary flips back, a bridge this call started is stopped, and a parked flag it cleared is restored. Promoting the current primary is a no-op (changed: false), so retrying after any failure is safe.

Parked, not deleted. The row, its vault key and its launch record all stay. For the legacy row the engine owns the flag, so REMOTE_TASK_TOKEN can stay set:

• the launchd wrapper checks legacy-parked before each launch; while the row is parked it launches nothing and raises no owner alert, but keeps the agent-bridge supervisor running, and it resumes by itself after un-park;
• the bare launch in startup-runtime.sh skips the bridge with a ⊘ line;
• the bridge itself asks connections.legacy_parked under the record lock right after it takes its lease (a new host-injected STAND_DOWN_GATE) and exits 75 when parked, so a launch that raced the promote never polls.

Named parked rows are skipped by launch_plan, plan_entry and start_added. unpark(instance) clears the flag and starts a named row's bridge as a secondary. The legacy row's launchers resume it. set_primary on a parked row is refused with parked_cannot_be_primary; reconnect also clears the flag.

API and CLI shape

• Module (src/connections.py):
  • promote(instance, *, repo, logs_dir, environ, workspace, vault, spawn, kill, python) returns the row plus previous (the parked instance, "" for legacy, None if nothing was parked), stopped_pid, legacy_stopped_pid, launch, changed and relaunch_needed.
  • unpark(instance, *, repo, logs_dir, ...) returns the row plus launch.
  • legacy_parked(workspace) returns a bool.
  • New refusal codes: legacy_bridge_not_running, new_primary_not_started, new_primary_not_connected, promote_conflict, parked_cannot_be_primary.
• CLI (scripts/connections.py):
  • promote INSTANCE --repo DIR --logs-dir DIR
  • unpark INSTANCE --repo DIR --logs-dir DIR ('' names the legacy row)
  • legacy-parked (exit 0 when parked, else 1)
• Runtime API: connection.promote {instance} returns {connection, previous, stopped_pid, launch, changed, relaunch_needed}. connection.unpark {instance} returns {connection, launch}.
• HTTP: POST /api/v1/connections/<instance>/promote and POST /api/v1/connections/<instance>/unpark, behind the same local API gate. Refusals are 409 with the code. The fixed segment legacy addresses the legacy row (instance "", which an empty path segment cannot carry), e.g. POST /api/v1/connections/legacy/unpark; legacy is reserved as a slug.
• Row: an optional parked boolean. status_for spells it out, and the derived state is parked with reason parked. status_for also carries legacy: true for the legacy row.
• Record: an optional top-level legacy_parked boolean for a rowless legacy bridge.

Where to look first

1. src/connections.py: promote, _promote_legacy, _await_connected, _undo_promote, _owner_row, unpark, legacy_parked, and the parked handling in launch_plan / plan_entry / derive_state.
2. src/launchd/gateway-bridge-wrapper.sh (the parked wait loop) and src/startup-runtime.sh (the bare-launch skip).
3. packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py: the STAND_DOWN_GATE hook, called after the lease heartbeat starts and before the proactive room is settled. src/remote-gateway-bridge.py injects it for the legacy, non-seat bridge only.

Behavior this proves (before/after)

Scripted repro, run in a temp workspace: the legacy row is primary and its bridge holds a real lease (a production-writer lease child), and s.space is added.

At the parent commit (e50b019):

legacy bridge running, pid 70389 | primary: ''
set_primary -> legacy_primary_running: the legacy unsuffixed bridge (pid 70389) is running and may carry the proactive room, which it decides only when it starts
promote -> not available on this commit

At this head:

legacy bridge running, pid 69002 | primary: ''
set_primary -> legacy_primary_running: the legacy unsuffixed bridge (pid 69002) is running and may carry the proactive room, which it decides only when it starts
promote -> primary: 's-space' | previous: '' | stopped pid: 69002 | new bridge pid: 69006
legacy row: parked parked | legacy_parked: True | plan: ['s-space']
owns unaddressed: s-space True | legacy False
unpark '' -> legacy_parked: False

Tests

All of these were run locally and pass:

• python3 tests/connections.test.py: 241 tests OK. The new Promote class covers:
  • the primary flipping before the new bridge spawns, with REMOTE_PROACTIVE_ROOM in its env;
  • an already-running secondary being stopped and relaunched as primary with the room;
  • relaunch_needed on a promote back to the legacy row;
  • the old bridge surviving (and the primary flipping back) when the new one never connects, a 401 failing fast, and a stopped secondary restarting on rollback;
  • a primary moved by another writer during the connect wait raising promote_conflict, with that move standing and nothing stopped or parked, and a parked target keeping its flag on a conflict;
  • a legacy bridge that a named row covers being stopped and parked (and surviving a failed connect), then unpark '' clearing it;
  • the running legacy primary being stopped and parked, with the new bridge started before the old one stops;
  • a refused stop rolling back to the old primary and stopping the bridge the call started;
  • a retry after a refused stop completing, and a repeat being a no-op that spawns no second bridge;
  • a parked target whose stop is refused being parked again, then promoted back;
  • a failed start never touching the old primary;
  • the launch plan, plan_entry (including env-only) and start_added skipping parked rows, and a legacy bridge that takes its lease later reading legacy_parked under the lock;
  • set_primary refusing a parked row;
  • un-park of the legacy and named rows (idempotent, never moves the primary);
  • promoting the legacy row (refused while its bridge is down, and parking the named primary when it is up);
  • a disabled target refused before anything runs.
• python3 tests/remote-gateway-singleton.test.py: 24 OK. The bridge stands down with exit 75 when the gate gives a reason, after the heartbeat and before the room settles; it keeps running when the gate says nothing or raises.
• bash tests/gateway-bridge-wrapper-self-heal.test.sh: PASS. With the row parked, the wrapper stays alive, launches nothing, keeps the agent-bridge supervisor up and writes no alert, and it launches the bridge after un-park.
• bash tests/startup-runtime-connections-launch-plan.test.sh: 27 passed. A parked legacy row is not launched and the skip is reported; named lanes still launch.
• bash tests/gateway-bridge-wrapper-primary-room.test.sh: all ok.
• python3 tests/runtime-api-connections-view.test.py: 22 OK. connection.promote off a running legacy bridge, then connection.unpark "", then legacy_bridge_not_running as CONFLICT.
• python3 tests/agent-api-connections.test.py: all ok. The new routes need the bearer; promote, unpark, promote back and the no-op repeat over real HTTP, and POST /connections/legacy/unpark reaching instance "".
• python3 tests/connections-cli.test.py: 17 OK.
• Also: ruff check on the changed Python files, and bash scripts/review-checks.sh --diff <(git diff origin/main...HEAD) (PASS).

Edge cases checked

• A stop refused for the old primary when the rollback's own stop of the new bridge also fails: the row is left un-parked, so the flag never says "down" over a live bridge.
• The legacy bridge relaunched by its wrapper while a promote is stopping it: it blocks on the record lock in its gate, then sees parked and stands down. A relaunch that took the lease before the stop deadline makes the stop fail with bridge_still_running, and the promote rolls back.
• An unreadable record parks nothing, which matches the existing env-only fail-open.

Docs

docs/connections.md is updated: the parked field, the refusal table (three new codes; promote added to disabled_cannot_be_primary, bridge_still_running and bridge_lease_indeterminate), a new "Promote: one operation, old primary running" section, the state table, and the CLI, runtime API and HTTP surface lists. docs/catalog.json is already verified for today.

Risks and gaps

• Live path. This touches the bridge's start and the launchd wrapper. I did not run a post-restart round trip on a live host, because that would interrupt the owner's running bridges. Coverage is real-process lease tests and wrapper and startup harnesses. A live check before merge would be: on a terminal-run core with the legacy bridge up, add a second homeserver, run scripts/connections.py promote <inst> --repo . --logs-dir <logs>, send a DM on the new homeserver and confirm the reply, then run unpark ''.
• Both bridges hold the room during the wait. For up to 60 s both bridges have REMOTE_PROACTIVE_ROOM in their environment. Only the one owns_unaddressed names posts there.
• Promote latency. A promote now waits for the new bridge's first gateway poll, which can take up to the bridge's long-poll window (about 35 s) when no task is queued. The HTTP server is threaded, so other routes are not blocked.
• Promote back to the legacy row. Its launchers own it, so the engine does not restart it; the answer says relaunch_needed instead.

🤖 Generated with Claude Code

[github-actions]

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 94%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

• packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py (100%)
• scripts/connections.py (100%)
• src/agent-api.py (100%)
• src/connections.py (98.5%): Missing lines 798,1275,1308
• src/remote-gateway-bridge.py (100%)
• src/runtime-api/connections_view.py (100%)
• src/runtime-api/dispatcher.py (100%)

Summary

• Total: 255 lines
• Missing: 3 lines
• Coverage: 98%

src/connections.py

Lines 794-802

  794     if not isinstance(removed, list) or not all(isinstance(x, str) for x in removed):
  795         raise SchemaError(f"connections.json: '{REMOVED_KEY}' is not a list of names")
  796     parked = payload.get(LEGACY_PARKED_KEY, False)
  797     if not isinstance(parked, bool):
! 798         raise SchemaError(f"connections.json: '{LEGACY_PARKED_KEY}' is not a boolean")
  799     return {"rows": rows, REMOVED_KEY: sorted(set(removed)), LEGACY_PARKED_KEY: parked}
  800 
  801 
  802 def _read_record(path: Path) -> dict:

---

Lines 1271-1279

  1271     with locked_file(_lock_path(workspace), create_mode=0o600):
  1272         rec = _read_record(path)
  1273         row = _find(rec["rows"], "")
  1274         if row.get("disabled"):
! 1275             raise DisabledCannotBePrimary("")
  1276         if not bridge_pid("", workspace):
  1277             raise LegacyBridgeNotRunning()
  1278         was_parked, was_primary = bool(row.get("parked")), row["primary"]
  1279         old = _owner_row(rec["rows"], workspace)

---

Lines 1304-1312

  1304                 return
  1305             if derive_state(sidecar, {})[1] == "auth_rejected":
  1306                 raise NewPrimaryNotConnected(instance, "auth_rejected")
  1307         if not pid or _exited(pid):
! 1308             raise NewPrimaryNotConnected(instance, "exited")
  1309         if time.monotonic() >= deadline:
  1310             raise NewPrimaryNotConnected(instance, "timeout")
  1311         time.sleep(0.2)

---

[sonichi]

Changes needed at 8b487118 (agent-assisted, Blue for Chi's seat). The locking and parking are solid. One blocker: after a promote, no bridge carries the proactive room.

Blocker: the new primary's bridge starts without REMOTE_PROACTIVE_ROOM.

• promote calls start(instance, …) (src/connections.py:1162) while the row is still a secondary.
• The primary flip happens only later, under the lock (:1184-1185).
• launch_env sets the room only when entry.get("primary") (:1560), so the new bridge launches with an empty room. Then the old bridge, the only one that had the room, is stopped and parked.
• Unlike set_primary, the result carries no relaunch_needed, so no caller knows to fix it. That breaks the "exactly one bridge carries the room" invariant in docs/connections.md.
• Proactive delivery stops until the next full restart.
• Promoting back to the legacy row has the same issue. That bridge was launched as a secondary, with its room stripped by the wrapper, and promote skips the LegacyPrimaryRunning guard that set_primary uses for exactly this case.
• Evidence: a background reviewer ran a probe at this head. With REMOTE_PROACTIVE_ROOM injected, promote returned primary=True, relaunch_needed=None, and the spawned bridge's room was ''. I confirmed the ordering by reading the code; my own re-run of the probe couldn't import the test helpers.
• Fix: relaunch the new primary's bridge after the flip (and the legacy row's when promoting back to it), or at minimum return relaunch_needed. Add a test that asserts the room on the spawned environment.

Should-fix:

1. "Started" isn't "connected". launch.ok with lease == "held" (:1160-1167) means the new bridge took its local lease, not that the gateway accepted its token. If the new token then gets a 401, the old bridge is already stopped and parked, and the owner has no working primary. Before stopping the old bridge, wait with a deadline for the new bridge's status to show connected, and roll back otherwise.
2. A legacy bridge can survive a promote. When a named row already covers the same server, import_from_env (:1342-1346) gives the env token no row. _owner_row (:1127) then can't see a running legacy bridge, so promote neither stops nor parks it. If it kept the room at startup, two bridges send nudges.

Nit: the wrapper's post-exit legacy-parked → continue check (gateway-bridge-wrapper.sh:217) is untested. Deleting it still passes gateway-bridge-wrapper-self-heal.test.sh.

What holds (verified by reading):

• One lock hold: stop, park and flip happen under a single hold of the record lock, and legacy_parked takes the same lock.
• The old bridge really stops: it goes through _stop_row and covers alias names.
• Parked rows stay down: they are skipped everywhere a bridge could start, including launch_plan, plan_entry, env_only (via _refuse_down), start_added, startup-runtime.sh:540, the wrapper and the bridge's stand-down check.
• Edge cases: a racing disconnect is refused and rolled back, a repeated promote is a no-op, and a disabled row is refused.
• HTTP: a Refusal maps to 409, and the new two-segment POST doesn't collide with existing routes.
• Earlier risk areas: none of the five from the prior connections review are reintroduced.

Tests run by the reviewer at this head:

• connections.test.py 231 OK; runtime-api-connections-view.test.py 22 OK; agent-api-connections.test.py PASS
• connections-cli.test.py 17 OK; remote-gateway-singleton.test.py 24 OK
• gateway-bridge-wrapper-self-heal.test.sh PASS; startup-runtime-connections-launch-plan.test.sh 27/0
• Mutation: replacing _stop_row with stopped = 0 fails 8 Promote tests.

Code-owner note: this touches src/runtime-api/, which only qingyun-wu and keweichen own, so it needs one of them in addition.

[sonichi]

Approve at 5b7fe5a9 (agent-assisted, Blue for Chi's seat). All three findings are fixed, and I checked each one by running the tests.

• Blocker, fixed.
  • The flip (r["primary"] = r is row) now happens in the first transaction, before start(), so launch_env gives the new bridge REMOTE_PROACTIVE_ROOM.
  • A running secondary is stopped and relaunched as primary.
  • relaunch_needed is now reported for the already_running and alias_running cases, and for promoting back to legacy.
  • Mutation check: I removed that early flip and re-ran connections.test.py. Exactly the two new ordering tests fail (…flips_before_the_new_bridge_launches… and …running_secondary_is_relaunched…).
• (a), fixed.
  • _await_connected waits up to 60 s for this pid's own sidecar to report connected.
  • It fails immediately on auth_rejected, and on exit or timeout it raises NewPrimaryNotConnected (409).
  • _undo_promote and _restart_secondary restore the old primary, the parked flag and a secondary that was stopped for the relaunch.
  • The old bridge is stopped only after this, under the lock.
• (b), fixed.
  • When no legacy row exists, the legacy bridge is stopped and the record-level legacy_parked is set.
  • The covered-legacy case stays up when the new bridge never connects, and that is tested.
• Ran at this head:
  • connections.test.py 239 OK; runtime-api-connections-view.test.py 22 OK; agent-api-connections.test.py PASS
  • connections-cli.test.py 17 OK; remote-gateway-singleton.test.py 24 OK
  • gateway-bridge-wrapper-self-heal.test.sh PASS; startup-runtime-connections-launch-plan.test.sh 27/0

Note, not blocking: for the up-to-60 s connect wait, both bridges carry the room. The old one still has it from its own launch, and the new one has it as primary. If proactive delivery goes through the outbox claim, a result is still delivered once. If anything posts nudges outside the outbox, it could double for that window. Worth one line in docs/connections.md, next to "exactly one bridge carries the room", so the transition isn't read as a violation.

Code owner: this still touches src/runtime-api/, so it needs qingyun-wu or keweichen as well.

[github-actions]

@cla-assistant check

[sonichi]

Re-approve at 559789d7 (agent-assisted, Blue for Chi's seat).

• Conflict guard. The second lock hold now raises PromoteConflict (a Refusal, so 409 promote_conflict) in three cases:
  • the row is gone or no longer primary;
  • the row to park was removed;
  • the row to park was already parked by another writer.
• Rollback on conflict. _undo_promote(..., restore_primary=False) stops only the bridge this call started and leaves the other writer's primary standing. _restart_secondary then brings back a secondary this call had stopped for the relaunch. Every other failure still restores before.
• Docs. docs/connections.md now states that both bridges hold REMOTE_PROACTIVE_ROOM during the connect wait (up to 60 s), and that only the one owns_unaddressed names posts there. That answers my note: the env overlap doesn't mean double posting.
• Ran at this head:
  • connections.test.py: 241 OK, including the two new conflict tests (a primary flip, and a third-party park during the wait).
  • runtime-api-connections-view.test.py: 22 OK.
  • agent-api-connections.test.py: PASS.
  • connections-cli.test.py: 17 OK.

Code owner for src/runtime-api/: qingyun-wu.