Skip to content

fix(health): status readers follow the primary connection, not the legacy sidecar - #5983

Merged
qingyun-wu merged 6 commits into
mainfrom
fix/primary-aware-health
Oct 29, 2026
Merged

qingyun-wu merged 6 commits into
mainfrom
fix/primary-aware-health

Conversation

@qingyun-wu

@qingyun-wu qingyun-wu commented Oct 28, 2026 •

Copy link
Copy Markdown
Collaborator

What changed, and why

After a move from ag2.space to s.space, the s.space row is primary and the legacy ag2.space row is parked. Several readers still assumed the legacy unsuffixed bridge was the install's gateway:

Reader (origin/main) What it did Now
src/health-check.py check_gateway_bridge and the sidecar readers _gateway_last_ok_age_h, check_runtime_identity, _gateway_status_stale_age_s, _gateway_status_ts_malformed, _gateway_serving (each defaulted to gateway-status.json) read the parked legacy bridge's stale sidecar as the primary and warned "…ag2.space mobile messages may not be delivered" read the primary row's sidecar (connections.sidecar_name(primary.instance)). Lane scans skip the primary's own lane and every parked or disabled row's lane, found by walking the rows with status_for. Messages name the primary's server, and lanes read as lane (server)
_gateway_configured (health-check), _gateway_configured (src/runtime-health.py), navigateUiAvailable (skills/ag2space-voice/navigate.ts) true only for the legacy env key or channels/ag2space/.env, so a fresh s.space-only install read as unconfigured also true when any enabled row exists (neither disabled nor parked), through one helper, connections.enabled_rows. The TS side asks it through the CLI (scripts/connections.py enabled) instead of copying the rule
src/dotspace.py current_server → default_link_via / default_cloud_origin fell back to LEGACY_SERVER an install with no AGENT_MXID linked via ag2.space callers pass primary_server=; dotspace stays stdlib-only and reads no record. Vendored copy synced with tools/sync_from_src.py (drift test green). src/ag2space_links.py passes it, behind a guarded import so a skill without core still falls back
scripts/sutando-whoami.sh, src/runtime-api/rundir.py enrolled_agent_id reported the device env / enrollment record's agent the primary row's agent_mxid first, then the old sources

Primary-row helpers in src/connections.py, for other connections work to reuse: this branch uses the read-only primary_row(workspace) that #5980 added on main, and adds read_rows(workspace) (every row, read-only, never imports from the environment, unlike list_connections), primary_server(workspace), enabled_rows(workspace) and sidecar_name(instance). _sidecar now uses sidecar_name. None of them touches launch_env or room-to-connection resolution.

The user-facing health-check wording became server-neutral here, not in the docs PR, because this PR rewrites those lines: GATEWAY_DOWN_DETAIL, the not-serving/stale details, and "gateway not configured on this host". The health-check fragments in tests/dotspace-literal-guard.test.py's allowlist are now unused. They are left in place so this PR does not conflict with the docs PR's edit to the neighbouring allowlist line.

Where to look first

src/health-check.py (new _gateway_* helpers before GATEWAY_STATUS_MAX_AGE_S, the default-path readers, the lane skips), then src/connections.py (helpers after primary_instance), then src/dotspace.py current_server.

Evidence (before / after)

A moved install: s.space primary, ag2.space legacy row parked with a day-old gateway-status.json, a fresh gateway-status.s-space.json, no legacy token anywhere. Same script against origin/main and this branch:

== origin/main
configured (s.space-only rows, no legacy token): False
gateway-bridge: {'status': 'warn', 'detail': 'process running but its status sidecar has not been written for 86400s (freshness window 180s) — the bridge is not reporting poll outcomes, so ag2.space mobile messages may not be delivered'}
== this branch
configured (s.space-only rows, no legacy token): True
gateway-bridge: {'status': 'ok', 'detail': 'running + connected (s.space)'}

The new suite at origin/main: FAILED (failures=3, errors=13). At HEAD: OK (16 tests). Mutation check: reverting only the sidecar default path to gateway-status.json fails the moved-install, silent-primary and three-server cases.

Tests

New, tests/health-check-gateway-primary-connection.test.py:

  • moved install: parked legacy row with a stale status file gives no warning. A parked named row is no stalled lane. A silent primary is still reported by its server. check_runtime_identity reads the primary sidecar.
  • s.space-only: health-check and runtime-health count it as configured. connections.py enabled exits 0 with s-space. Parked or disabled rows alone configure nothing (exit 1).
  • three servers (ag2.space parked, s.space secondary, dev.s.space primary): dev.s.space is read as the primary and s.space as a lane (lane s-space (s.space)). primary_server drives current_server and default_link_via.
  • read helpers never import from the environment, and an unreadable record reads as none. current_server precedence: mxid, then gateway, then primary, then "", and an invalid primary is ignored. default_cloud_origin(primary_server=) stays within configured entries. Links use the primary. rundir reads the primary, then the enrollment record, and falls back when the record raises.

Updated: tests/ag2space-voice-navigate.test.ts (connection-only install offers navigation. enabledConnectionExists against a real record: no record, parked plus disabled, s.space-only), tests/sutando-whoami.test.sh (primary row's agent wins), tests/health-check-gateway-bridge.test.py (pins an empty workspace so the host's own rows never leak into the legacy-token cases).

python3 tests/health-check-gateway-primary-connection.test.py    OK (16)
npx tsx --test tests/ag2space-voice-navigate.test.ts             pass 38/38 (the new cases included)
bash tests/sutando-whoami.test.sh                                PASS
274 suite files matching health-check* / connections* / dotspace* / runtime-* / ag2space-* / gateway-* / ag2-sparrow-drift / util-paths-runtime* / core-health* / skill-usage*, plus whoami and rundir-resolver: 272 pass, 2 fail (both below)
npm run typecheck && npm run typecheck:skills                   rc 0
npx eslint (changed TS)                                          rc 0
bash scripts/review-checks.sh --diff <(git diff origin/main...HEAD)   PASS
python3 tests/hermetic-env-structural.test.py                    OK
python3 scripts/hermetic-workspace-guard.py run <new suite>      OK
python3 scripts/lint-hermetic-bridge-tests.py                    ok
python3 skills/release/scripts/docs_audit.py                     PASS
diff-cover (coverage over the touched suites) vs origin/main     95%+: connections, dotspace, ag2space_links, runtime-health, scripts/connections.py 100%; rundir 100% after the added fallback case

tests/health-check-fix-down-bridges.test.py case f fails identically on origin/main on a host whose own Slack bridge is running. That is a pre-existing host dependence and not touched here. tests/runtime-api-request-expiry.test.py failed once during the long batch run and passes when run alone (rc 0). In CI, tests/remote-gateway-bridge.test.py (python shard 1/9) also fails on main at b72b5ca, before this PR.

Feature/documentation consistency

docs/connections.md documents the read-only helpers, the enabled CLI verb and what the health check reads. docs/dotspace.md documents primary_server=. docs/catalog.json dates are bumped.

Risks / edge cases

  • Runtime identity moves with the primary. rundir.enrolled_agent_id now prefers the primary row's agent, so the per-actor run dir key follows a primary flip. A runtime daemon started before the flip keeps its old socket until it restarts. util_paths' canonical actor uses the same function, so the two sides agree.
  • cloud_auth.DEFAULT_CLOUD_ORIGIN is deliberately unchanged. For every registered server without its own Cloud the answer is the same, and changing the credential owner's default deserves its own review.
  • Without a connections record, every reader behaves exactly as before (legacy sidecar, legacy tokens).

Follow-ups

🤖 Generated with Claude Code

…gacy sidecar

After a move (s.space primary, the ag2.space row parked) the legacy
gateway-status.json goes stale by design, and the gateway-bridge check read it as
the primary and warned that messages may not be delivered. A fresh install whose
only gateway is a connection row read as unconfigured.

- connections: read-only helpers read_rows / primary_row / primary_server /
  enabled_rows and sidecar_name; CLI verb `enabled`.
- health-check: the primary row's sidecar is the gateway's own; lane scans skip
  the primary's lane and parked/disabled rows (via status_for); messages name the
  server instead of a hardcoded one.
- "is a gateway configured" (health-check, runtime-health, navigate_ui) is also
  true when any enabled row exists; the voice tool asks the CLI, not a copy.
- dotspace: current_server / default_link_via / default_cloud_origin take the
  caller's primary_server before the legacy fallback (vendored copy synced).
- whoami and rundir report the primary row's agent_mxid first.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
qingyun-wu and others added 3 commits October 28, 2026 12:22
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ter in the CLI-backed probe test

main gained an identical read-only primary_row, so this branch's copy is dropped.
A Linux runner's /usr/bin/python3 reads as the macOS stub to resolvePython, so the
test names its interpreter through SUTANDO_PY.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Requesting changes: the health-check readers now follow the primary row, but two sibling readers of the same sidecar are left on the legacy file.

  • Bandaid, sibling sites: src/core-input-watch.py:529 (_gateway_status, which feeds gateway_alive() and so the core's gateway-down state) and src/services_status.py:73 (GATEWAY_STATUS_PATH, the desktop services row) still hard-code gateway-status.json. On the moved install this PR targets, that file is the parked bridge's stale sidecar, so both always read "no opinion" and fall back to the identity-blind pgrep their own docstrings call a proxy for the wrong thing. connections.sidecar_name(primary.instance) added here is the fix for both. Please move them too, or file an issue and name them under Follow-ups.
  • Nit, non-blocking: _gateway_lane_label(ln) without rows (src/health-check.py:6956, :6958, :6966) runs _gateway_rows(), which calls status_for (sidecar read plus lease probe) on every row, once per lane. _gateway_lanes_not_served also needs only the raw primary/disabled/parked flags, so read_rows would do.
  • Checked, fine: disabled rows derive state == "parked" (derive_state), so the lane skip covers both, as the body says. Tests import and exercise the real modules, with no source-text assertions. The new suite and health-check-gateway-bridge pass locally at head. The SUTANDO_WORKSPACE in tests/sutando-whoami.test.sh is a test-mode fixture, not a default.
  • CI: python shard 1/9 fails on tests/remote-gateway-bridge.test.py. It also failed on main at b72b5ca, and main fixed it in 9826add (#5984). A rebase should clear it. license/cla is present.
  • There are no approvals yet.

— Sutando (susans-MacBook-Pro)

qingyun-wu and others added 2 commits October 28, 2026 15:39
…primary connection's sidecar

One resolver, connections.primary_sidecar_name, now serves health-check, core-input-watch and
services_status. Per-lane labels and the lane skip read stored rows only, with no sidecar read or
lease probe per lane.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Addressed in 8881b56 (main merged in):

  • Sibling sites: core-input-watch (_gateway_status, so gateway_alive()) and services_status (the services row; GATEWAY_STATUS_PATH replaced by gateway_status_path()) now read the primary's sidecar. All three readers share one resolver, connections.primary_sidecar_name(state_dir), and health-check's own helper delegates to it. A sweep of gateway-status readers found no others needing the primary: the rest are the bridge writer, instance-aware hatch readers, and glob-all-lanes scans.
  • Nit: the lane label and lane skip now use stored rows (read_rows plus the raw primary/parked/disabled flags), with no sidecar read or lease probe per lane.
  • Tests cover both readers against a moved install, the unreadable-record fallback, and the no-probe rows.

@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@github-actions

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 94%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • scripts/connections.py (100%)
  • src/ag2space_links.py (100%)
  • src/connections.py (100%)
  • src/core-input-watch.py (100%)
  • src/dotspace.py (100%)
  • src/health-check.py (100%)
  • src/runtime-api/rundir.py (100%)
  • src/runtime-health.py (100%)
  • src/services_status.py (100%)

Summary

  • Total: 100 lines
  • Missing: 0 lines
  • Coverage: 100%

@qingyun-wu
qingyun-wu merged commit 4908f89 into main Oct 29, 2026
42 checks passed
@qingyun-wu
qingyun-wu deleted the fix/primary-aware-health branch October 29, 2026 00:15
bassilkhilo-ag2 added a commit that referenced this pull request Oct 29, 2026
main gained its own answer to "which gateway-status sidecar do the readers
read": #5983 routed health-check, core-input-watch and services_status through
connections.primary_sidecar_name (the primary connection row's suffixed file,
the unsuffixed one without a primary row), and #5971 / #6008 reworked the
neighbouring health-check text and the configured-gateway probe. This branch
routed the same three readers through gateway_serving.status_path, the
desktop's active-lane selector. Both are kept: the active lane wins when the
selector names a valid lane whose file exists (an s.space-only desktop install,
where s.space is a secondary lane and no row is primary), else the primary
connection's sidecar as main resolves it.

Conflicted files and how each was resolved:

- src/gateway_serving.py (auto-merged, edited): status_path's fallback is now
  connections.primary_sidecar_name(state_dir) via a lazy import instead of the
  literal gateway-status.json, so one resolver carries both rules and no reader
  composes them privately. Without a primary row that name is the unsuffixed
  file, so the branch's fallback cases are unchanged.
- src/core-input-watch.py: _gateway_status reads gateway_status_path(state_dir)
  (this branch); main's private _primary_sidecar_name helper is dropped because
  its logic now lives in the shared resolver; gateway_alive's docstring names
  both sources.
- src/services_status.py: main's no-argument gateway_status_path(), which
  services-status.test.py pins, is kept and returns status_path(STATE_DIR);
  the import alias becomes _lane_status_path so the function no longer shadows
  it (the raw auto-merge had the def shadowing the import). Main's registry
  comment is kept with "active lane, else primary" wording; the probe calls
  gateway_status_path().
- src/health-check.py: all five default-path sites keep main's
  _gateway_primary_sidecar() (pinned by health-check-gateway-primary-connection
  .test.py); that helper now resolves the name through status_path and still
  passes it through status_read_path for the legacy workspace-root fallback.
  This branch's _gateway_status_path helper is removed as redundant.

Tests run at this commit, all green: gateway-status-active-lane,
gateway-serving-shared-owner, health-check-gateway-bridge,
health-check-gateway-primary-connection, health-check-sspace-migration,
health-check-gateway-lane-verdict, health-check-gateway-stale-lane,
health-check-gateway-transient-outage, health-check-gateway-outage-duration,
health-check-gateway-probe-unknown, health-check-notify-gateway,
gateway-connected-requires-last-ok, gateway-age-helpers-overflow-safe,
gateway-runtime-identity, core-input-watch-gateway-probe, core-input-watch,
services-status-liveness-patterns, services-status, core-health-verdict,
util-paths-runtime-identity-absent.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants