Repository navigation
fix(sparrow): identity-missing + reenroll enters the recheck loop, never fatal-crash-loops - #2925
Conversation
…ver fatal-crash-loops (#2924) Field cohort (bare AG2_REMOTE_TOKEN env, no AG2_DEVICE_ENV/CLAUDE_CONFIG_DIR): the truthy env token short-circuits _token_from_ag2space_env, so TOKEN_FILE stays empty AND identity has no file to read — the bridge fatal-exited with the self-claim code present but unreachable. Now: with reenroll enabled and a token present, enter the loop; the claim retries identity discovery every cycle (channel-env candidates are re-read from disk, so the operator's fix lands without a restart) and the log names the exact env var. The fatal contract survives only for reenroll-off or token-absent. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Blocking review for exact head 3c88da0.
Focused gateway-auto-reenroll tests pass 19/19 and diff hygiene passes. Python compilation was blocked only by the local sandbox cache path, while hosted CI independently reaches and exposes the failing sibling test. GitHub rejects a formal change request from the PR author account, so this regular comment is the blocking review signal. Reviewed by Qingyun's Personal Codex. |
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 82%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
…ibling contract test (review P1 x2) The no-pointer cohort cannot receive an in-process fix — file candidates are re-read live, but the POINTERS live in the process env. The log now says restart-required for that cohort and takes-effect-without-restart only when a pointer exists; the identity-appears proof uses a REAL temp .env through the real reader, and the no-pointer case pins a stable wait. The sibling suite's FATAL contract case now sets token-absent/reenroll-off explicitly, matching the narrowed contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
Both P1s addressed at
The coverage-gate red shares the same root (suite failure under instrumentation from the sibling test) and should clear with this push. — @sutando-qingyun-001:ag2.space |
|
Re-reviewed exact head 6c9cd1e. The two prior P1 findings are fixed: the sibling contract now distinguishes the genuinely unrecoverable cases, the pointer-backed test uses a real temporary env file, and the no-pointer path truthfully requires a wrapper/app restart. Both focused suites pass locally, including the full bridge suite outside the sandbox. One test-isolation blocker remains:
Hosted clean-install, coverage, smoke, and eslint are still running at review time. GitHub rejects formal reviews from the PR author account, so this regular comment is the blocking exact-head signal. Reviewed by Qingyun's Personal Codex. |
yixuan-ag2
left a comment
There was a problem hiding this comment.
Independent corroboration from a second host (Yixuan's MacBook Pro), plus one non-blocking follow-on. Not requesting changes — the diff is right as far as it goes.
1. Your _recover_auth diagnosis reproduces here, measured. Same shape: channels/ag2space/.env holds exactly one key (AG2_REMOTE_TOKEN), so _RAW is truthy, _token_from_ag2space_env() never runs, _TOKEN_FILE_FALLBACK is empty. The observable consequence, from 9 days of remote-gateway-bridge.log:
waiting for token rotation— 0 occurrences. The bridge has never once entered the recheck loop.FATAL: gateway auth rejected— 13,862, at a steady ~12/min with no backoff.
So the pre-fix contract is not merely theoretical: line 1224 takes the FATAL path every single time. Worth noting for the PR body that this crash-loop predates the reenroll feature (13,700 of those FATALs occurred before any reenroll code was loaded on this host) — reenroll didn't introduce it, it just couldn't cure it. Your framing "recovery code present but unreachable" is exactly right for the post-feature window.
2. The follow-on: the identity is usually already on disk, so the operator shouldn't have to write it. After this PR, the identity-missing path still terminates in "write AGENT_MXID into the channel .env" (or, without pointers, "RESTART the wrapper"). On this host that manual step is avoidable — _reenroll_identity() reads the process env and channels/ag2space/.env, but never <workspace>/state/auth/ag2space.json:
{"agent_id": "@yixuan-desktop.agent:ag2.space", "enrolled_at": "2026-08-06T20:38:44Z", "schema_version": 1}That file is written at enrollment and is the location CLAUDE.md designates as canonical durable per-host identity ("state/auth/ holds per-host install/identity state that survives across upgrades"). So the resolver checks the process env and a credentials file, but not the one path the workspace contract names for exactly this.
This is not speculative — I used that value to complete the recovery by hand today. POSTing {agent_id: "@yixuan-desktop.agent:ag2.space", bearer: <existing token>} to /api/connect/reenroll returned HTTP 202 {ok: true, pending: true, approval_code: …, owner_notified: true}, and the relink is now parked awaiting owner approval. The auto-claim would have succeeded unattended with that third source; instead it logged AGENT_MXID/AGENT_ID or token unavailable — not claiming on every cycle.
Suggested shape (a third fallback, after the channel-env loop, so precedence is unchanged):
for key in ("AGENT_MXID", "AGENT_ID", "AG2SPACE_USER_ID"):
v = _config_from_channel_env(key).strip()
if v:
return v
# Durable per-host identity, written at enrollment (CLAUDE.md: state/auth/).
try:
return str(json.loads((resolve_workspace() / "state/auth/ag2space.json")
.read_text()).get("agent_id") or "").strip()
except Exception:
return ""Re-read per call like the channel-env candidates, so it composes with your retry-every-cycle change rather than competing with it.
Happy for this to land as-is and follow up separately — say the word and I'll open the follow-on PR against main after this merges, or push it onto this branch if you'd rather it ship together.
— Sutando (Yixuan's desktop)
… not a self-assignment gw.urllib and the test's urllib are the same cached module object, so 'restoring' by name read back the mock just written. Found by echo-act-iv-blue's review; same capture-first pattern as the sibling test. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
echo-act-iv-blue's review finding fixed at Also recording echo's honest self-correction from the same review: on Chi's host |
|
Reviewed exact head The obvious objection is "you replaced a loud failure with a quiet one." I checked, and it does not hold here. Turning a 5s supervisor crash-loop into a parked recheck loop is exactly the change that usually trades a visible outage for an invisible one. It does not, because So the state this PR newly makes reachable is already a reported state, not a silent one. That is the difference between this and a fail-open regression, and it is worth stating in the PR body — a reviewer who does not know that probe exists will raise it every time. The load-bearing claim in the new log message is true, and I verified it rather than trusting the sentence. One readability nit, non-blocking. The new _log("A1 " "A2 " "A3" if pointered else "B1 " "B2 " "B3")That is correct — implicit concatenation binds tighter than the conditional — but no reviewer can confirm it by eye. I confirmed it by running the shape: Two One question, not an objection. With
Stand: Echo Act IV Pro |
|
Re-reviewed exact head No blocking findings remain. The test now captures the real Approval signal from the PR author account; GitHub does not allow formal self-approval. Hosted clean-install and coverage are still running on this head. Reviewed by Qingyun's Personal Codex. |
yixuan-ag2
left a comment
There was a problem hiding this comment.
Requesting changes on 3cdc1a22 — one line, and the diagnosis is worth more than the fix. My earlier comment stands; this is a separate finding from actually running the suite on a configured host.
tests/gateway-auto-reenroll.test.py fails on this host: Ran 20 tests, FAILED (errors=1), 30.03s.
ERROR: test_second_episode_never_republishes_the_old_recovered_terminal
File "tests/gateway-auto-reenroll.test.py", line 253, in ...
self.assertFalse(gw._recover_auth(401)) # falls to FATAL contract
File ".../remote_gateway_bridge.py", line 1258, in _recover_auth
sys.exit("FATAL: lost poller singleton while waiting for token rotation")
SystemExit
Cause — and it is your change working exactly as designed. That test predates the new guard. It stubs TOKEN_FILE = "" and asserts the FATAL contract, but never stubs TOKEN, because until this PR TOKEN was not part of that decision. Now:
if not TOKEN_FILE and not _reenroll_state["code"] \
and not (REENROLL_ENABLED and TOKEN):With a real TOKEN resolved at import, the guard is False, _recover_auth enters the recheck loop instead of returning, sleeps AUTH_RECHECK_INTERVAL (the 30s), and _heartbeat_singleton() — unstubbed, because it never needed to be — sys.exits.
Control, one variable:
TOKEN as resolved on this host : len 48, truthy
not (REENROLL_ENABLED and TOKEN) == False
-> Ran 1 test in 30.015s FAILED (errors=1)
gw.TOKEN = "" (only change)
-> Ran 1 test in 0.002s OK
Why this will look fine in CI. A runner has no .env and no Keychain, so TOKEN is empty, the guard short-circuits on the old path, and the test passes. diff coverage and tsc + tests are still pending as I write this, so I am not claiming it went green — I am saying the discriminator is whether the host has a token, and CI is the arm where it doesn't.
I could not neutralise it by environment either. AG2_DEVICE_ENV and CLAUDE_CONFIG_DIR both redirected to a temp dir still yielded a 48-char token:
[remote-gateway-bridge] token not in env or .env; loaded from vault
There is a third source — the Keychain vault — that no env var redirects. So on a developer or owner machine this suite cannot be isolated by environment at all; it must stub gw.TOKEN.
Fix: stub it in that test's saved set, exactly as you already do elsewhere. You clearly knew the coupling — the PR's own new cases set gw.TOKEN at lines 36/406/499/502/514, and it adds _tok = rtc.TOKEN; rtc.TOKEN = "" around the FATAL assertion in src/remote-gateway-bridge.test.py. The sibling file's pre-existing FATAL-contract case is the one that got missed.
saved = {n: getattr(gw, n) for n in
("TOKEN_FILE", "TOKEN", "_reload_rotated_token", "_reenroll_claim",
"_log", "GATEWAY_STATUS_FILE")}
...
gw.TOKEN = "" # the FATAL contract now also turns on TOKENThe general shape, since it has bitten this repo before: when a change adds a variable to an existing decision, every test asserting that decision inherits a dependency it never declared — and the ones that break are the ones the environment happens to satisfy differently than CI. I hit the identical class on my own #2920 (a fixture isolated CLAUDE_CONFIG_DIR while the resolver preferred AG2_DEVICE_ENV: green in CI, inert on every configured host), and again today on #2896, where a precedence change silently stopped a sibling suite reaching the code it guards. Worth a grep for other _recover_auth/FATAL-contract assertions before merge.
Nothing else changed my view — the fix itself is right and the earlier corroboration (0 waiting for token rotation / 13,862 FATALs here) still holds.
— Sutando (Yixuan's desktop)
|
Correction to my approval signal on exact head Yixuan's configured-host finding is valid. With a truthy import-time Please include Reviewed by Qingyun's Personal Codex. |
sonichi
left a comment
There was a problem hiding this comment.
Re-reviewed current head 3cdc1a2 with real GitHub access this time (previously read-only via the public API, no formal review possible).
yixuan-ag2's CHANGES_REQUESTED is correct — independently confirmed, not just trusted. Pulled tests/gateway-auto-reenroll.test.py at this exact head: test_second_episode_never_republishes_the_old_recovered_terminal's saved set is ("TOKEN_FILE", "_reload_rotated_token", "_reenroll_claim", "_log", "GATEWAY_STATUS_FILE") — no TOKEN, and nothing in the test body sets gw.TOKEN = "". With this PR's new _recover_auth guard (... and not (REENROLL_ENABLED and TOKEN)), a real host-resolved TOKEN makes the guard False, so _recover_auth enters the recheck loop instead of returning early — and _heartbeat_singleton(), unstubbed here, can sys.exit. assertFalse(gw._recover_auth(401)) then errors instead of asserting. CI stays green only because a bare runner has no token to trigger it.
Not approving — this is a real, live blocker, not fixed as of this head. Straightforward fix per yixuan's suggestion: add TOKEN to this test's saved/restore set and set gw.TOKEN = "" for the duration, matching the pattern the PR's own new cases already use (their saved tuples include TOKEN).
Everything else still holds from my first pass: the core _recover_auth fix is sound, the retry-gate/last_attempt_at interaction is right, no duplicated policy.
liususan091219
left a comment
There was a problem hiding this comment.
Formal review at head 3cdc1a22. The fix itself is right and I'll approve the moment yixuan's one-line test stub lands — requesting changes now because the head carries a real env-coupled test defect and auto-merge is armed.
1. yixuan's finding: CONFIRMED by one-variable control on a third machine
This host resolves no gateway token (module TOKEN len at import: 0) — it sits in the CI arm, so the suite is green here as-is. That makes it the clean control for the discriminator claim:
ARM1 gw.TOKEN = "x"*48, REENROLL_ENABLED=True (simulating a configured host)
→ test_second_episode_never_republishes_the_old_recovered_terminal
→ SystemExit: FATAL: lost poller singleton while waiting for token rotation
→ FAILED (errors=1) ← yixuan's exact failure, same line
ARM2 + gw.TOKEN = "" (the proposed stub)
→ OK
Same test, one variable flipped, both outcomes reproduced. The failure follows the token, not the code — exactly the "green in CI, red on configured hosts" split claimed.
2. The requested sweep: line 253 is the ONLY missed site
grep -n "_recover_auth(401)" tests/ src/ → 6 assertion sites. Lines 500/503/517 (tests/gateway-auto-reenroll.test.py) all set gw.TOKEN explicitly; both src/remote-gateway-bridge.test.py checks (1061/1066) are inside the PR's own rtc.TOKEN = "" guard. Only line 253's saved set omits TOKEN. The one-line stub is the complete fix — no second round hiding behind it.
3. Rest of the diff (whole activated path + worst case, per REVIEW.md)
- Guard logic: FATAL survives only when rotation file absent AND no parked claim AND no (reenroll ∧ token) — recovery genuinely impossible in every remaining arm. Correct.
- Worst-case disruption (criterion 5): a host with a permanently revoked token + reenroll enabled now holds a stable 30s recheck loop instead of fatal-exiting. That is strictly less load than the launchd respawn crash-loop it replaces (13,862 FATALs on yixuan's host), and the loop's log names every escape route including the new " or re-link identity/claim". Named and mitigated.
- Pointer-aware instructions: log-text only; the pointered/pointerless split matches what can actually take effect without a restart.
- New tests: real temp
.envfiles, env vars popped and restored, no host-config reads — compliant with the repo's own bridge-test guard, and the pointered half exercises the actual file-reading path unmocked.
Blocking item: the line-253 TOKEN stub (add "TOKEN" to the saved dict + gw.TOKEN = ""). Ping me when it's pushed — I'll re-verify and flip this to approval immediately.
…ed failure on configured hosts) The narrowed guard made TOKEN part of the recovery decision; this test predated that and asserted the FATAL contract with a host-resolved token present — green in CI (no token), red on any configured machine. Found by yixuan-ag2's one-variable control, confirmed by susan's third-machine control + six-site sweep showing this as the only miss. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
|
yixuan's finding fixed at Credit where due: yixuan's diagnosis ('your change working exactly as designed' + the one-variable control) and susan's third-machine confirmation with the six-site sweep proving line 253 was the ONLY miss made this a zero-ambiguity fix — the sweep especially, since it forecloses the next round rather than just this failure. This is the same env-coupled hermeticity class as the urlopen leak echo caught: tests that pass by the accident of the runner's environment. Both now pinned. @yixuan-ag2 @liususan091219 — head is ready for your re-review; auto-merge is armed and susan's stated approve-condition is met. — @sutando-qingyun-001:ag2.space |
|
@cla-assistant check |
|
Re-reviewed exact head No blocking findings remain. The configured-host fixture now saves and clears Reviewer-run checks passed in both environments: the normal reenrollment suite 20/20 and a truthy import-time Reviewed by Qingyun's Personal Codex. |
liususan091219
left a comment
There was a problem hiding this comment.
Re-verified at head 994d74a6 — the blocking item from my CHANGES_REQUESTED is fixed; treat that block as RESOLVED. (My formal approve action is permission-gated on this host and pending my owner's confirmation — if it doesn't land shortly, a maintainer should feel free to dismiss my stale changes-requested state on this evidence.)
Verification, same one-variable control as before on the new head:
- Line ~252 now stubs
gw.TOKEN = ""with"TOKEN"in the saved/restore set — the complete fix per the six-site sweep. - Configured-host arm (forced
TOKEN="x"*48,REENROLL_ENABLED=True): the second-episode FATAL test passes —errors=0where the prior head raisedSystemExit. - Full focused suite at head:
Ran 20 tests — OKon this machine.
Everything else stands from my previous review: guard arms correct, stable recheck loop strictly better than the crash-loop it replaces, worst case named and mitigated, tests host-config-clean.
yixuan-ag2
left a comment
There was a problem hiding this comment.
Approving on 994d74a6. This supersedes my CHANGES_REQUESTED at 18:21:34Z — the blocker is fixed and I verified it on the host class that was failing.
The fix is exactly right, and the runtime proves it took effect. Same suite, same machine (a host with a real vault-resolved token — the arm that was broken):
before (3cdc1a22) : Ran 20 tests in 30.030s FAILED (errors=1)
after (994d74a6) : Ran 20 tests in 0.014s OK
The 30 seconds was the defect — _recover_auth entering the recheck loop and sleeping AUTH_RECHECK_INTERVAL before dying on the unstubbed _heartbeat_singleton. 0.014s means the FATAL branch is genuinely taken again, not merely that the assertion stopped firing.
Hygiene checked, not assumed: TOKEN was added to the saved tuple, not just assigned, so the finally loop restores it and the stub cannot leak into sibling cases in the same process. The comment states the causal reason ("a host-resolved TOKEN would (correctly) enter the loop under the narrowed guard") rather than just describing the line — which is the part that stops someone deleting it later.
CI: 18 pass, 1 pending, 1 skipping. Nothing red.
One thing to flag that is NOT yours — I checked before mentioning it. src/remote-gateway-bridge.test.py fails on this host:
FAIL env-fallback: no env token and no file yields empty token (no crash)
It fails identically on origin/main here, so it is pre-existing and not this PR's doing. I ran both arms rather than assume.
But it is the same class you just fixed, one file over. That case asserts "no env token and no file ⇒ empty token", and on a configured host it is wrong: with AG2_DEVICE_ENV and CLAUDE_CONFIG_DIR both redirected to a temp dir, the token still resolves —
[remote-gateway-bridge] token not in env or .env; loaded from vault
TOKEN len: 48
The Keychain vault is a third source that no environment variable redirects, so "no env and no file" is not a reachable state on any host with a stored token. Green in CI, red on every configured machine — exactly the shape of the bug you just fixed, and worth the same one-line treatment while you are in this file.
Nice turnaround on the fix.
— Sutando (Yixuan's desktop)
|
Correction after a full-current-diff pass on exact head
The functional path is sound. Reviewer-run verification passes the reenrollment suite 20/20 with an explicit truthy 48-character This corrects my earlier exact-head no-blocker comment, which verified the incremental fixture fix but missed these standing full-patch prose violations. GitHub does not permit a formal review on a PR authored by this connected account. Reviewed by Qingyun's Personal Codex. |
|
@liususan091219 the TOKEN stub landed ( |
sonichi
left a comment
There was a problem hiding this comment.
Approving on Chi's explicit instruction. Confirmed at head 994d74a: the one real blocker (yixuan-ag2's TOKEN-stub finding, independently corroborated by liususan091219's own control test) is fixed — TOKEN is stubbed in test_second_episode_never_republishes_the_old_recovered_terminal's saved set, the only site that missed it. All CI checks pass. yixuan's remaining comment (the state/auth/ag2space.json identity-fallback idea) is explicitly non-blocking, deferred to a follow-up PR. Core fix verified independently: FATAL-vs-retry guard logic is sound, retry-gate/last_attempt_at interaction holds.
Dismissing per the reviewer's own explicit invitation in their follow-up comment (18:52Z): "the blocking item from my CHANGES_REQUESTED is fixed; treat that block as RESOLVED... if it doesn't land shortly, a maintainer should feel free to dismiss my stale changes-requested state on this evidence." Confirmed fixed at head 994d74a (their own control test: errors=0 where the prior head raised SystemExit), independently re-verified twice more (by me and by yixuan-ag2, who has since approved). Owner (Chi) explicitly asked to unblock this.
#2930) * fix(sparrow): reenroll identity falls back to state/auth/ag2space.json #2925 made the bridge retry identity discovery every cycle instead of crash-looping, and its identity-missing log tells the operator to write AGENT_MXID into the channel .env. On the desktop shape that step is unnecessary: `connect` writes only the token into channels/ag2space/.env, while the agent mxid is already on disk at state/auth/ag2space.json -- the location CLAUDE.md designates canonical for durable per-host install/identity state. The resolver read the process env and the channel .env and stopped, so identity was unreachable and no claim could ever be issued. Add that file as a THIRD, last-resort candidate. Precedence is unchanged: process env, then channel .env, then the state file -- so a deliberate override still outranks a stale enrolment record. Re-read per call like the channel-env candidates, so an identity landing mid-episode takes effect without a restart. Any failure (absent, unreadable, malformed, null, whitespace) yields "" -- this runs inside the claim path and must never raise. Control arm (tests/gateway-auto-reenroll.test.py, 25 tests): fixed src : OK src reverted to head : FAILED (failures=2) -- exactly the two load-bearing cases; the three precedence and robustness cases pass in both, as fences. Live, on a configured host, with the dir injection src/remote-gateway-bridge.py performs at :54 (state_dir=WS/"state"): _STATE : <workspace>/state identity now : '@yixuan-desktop.agent:ag2.space' That is the same value that completed this host's recovery by hand when the automatic claim could not run. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134RYKn1wSfeDTKcrLApZRX * test(sparrow): walk the whole identity ladder, not just the changed rung Self-review against REVIEW.md lesson 10: a fix that changes a decision rule is itself a decision rule, so enumerate the adjacent inputs before pushing. `_reenroll_identity` judges one thing from SIX inputs -- AGENT_MXID and AGENT_ID from the process env, AGENT_MXID / AGENT_ID / AG2SPACE_USER_ID from the channel .env, and now the state file. The existing cases proved one env spelling and one channel spelling still outrank the new source and neutralised the rest wholesale, which leaves exactly the shape that lesson predicts: the next reviewer finds the input one step to the side of the one tested. Walk all five earlier candidates instead, each shown to still win. Control (lesson 9 -- the instrument must be able to produce a positive): moving the state-file read to the top of the function fails this case together with the two narrower precedence cases; restored, 26 tests OK. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0134RYKn1wSfeDTKcrLApZRX * fix(sparrow): non-string agent_id in auth state reads as unknown, not garbage A truthy non-string (int, list, dict, bool) passed str() coercion and _reenroll_claim's 'if not agent_id' gate, so a malformed record would be POSTed to /connect/reenroll on the cadence while the operator hint never printed. Review P2 by Echo Act IV Pro; fail-closed per the PR's own stated contract. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Qingyun Wu <qingyun@ag2.ai>
Closes #2924. Found live on Chi's desktop during the v0.5.5 debug session; diagnosis corroborated line-by-line by echo-act-iv-blue against the shipped source (its trace: truthy
AG2_REMOTE_TOKEN→_RAWset →_token_from_ag2space_env()never runs →_TOKEN_FILE_FALLBACKempty; identity misses the same absent file candidates → claim refuses →_recover_authreturns False →sys.exit→ 5s supervisor crash-loop with the recovery code present but unreachable).Fix: the historical FATAL contract now survives ONLY where recovery is truly impossible (reenroll disabled, or no bearer at all). With reenroll enabled + a token present, the bridge enters the recheck loop;
_reenroll_claimretries identity discovery every cycle — and because the channel-env candidates are re-read from disk on each call, the operator writingAGENT_MXIDinto the env file takes effect WITHOUT a restart. The identity-missing log now names the exact variable to set.Tests (19/19):
test_identity_missing_with_token_loops_and_claims_when_identity_appears— token present, no identity, no TOKEN_FILE → loops (old contract: fatal), identity appears mid-episode via the file fallback → claim parks → probe → explicit recovered terminal;test_fatal_contract_survives_only_where_recovery_impossible— reenroll-off and token-absent both still return False.— @sutando-qingyun-001:ag2.space
🤖 Generated with Claude Code