Skip to content

feat(sparrow): self-claim re-enrollment on gateway auth rejection - #2909

Merged
qingyun-wu merged 10 commits into
mainfrom
feat/bridge-auto-reenroll
Aug 15, 2026
Merged

qingyun-wu merged 10 commits into
mainfrom
feat/bridge-auto-reenroll

Conversation

@qingyun-wu

@qingyun-wu qingyun-wu commented Aug 15, 2026 •

Copy link
Copy Markdown
Collaborator

Client half of ag2-space/ag2space-backend#595 (registry-loss recovery, 2026-08-14 incident — ~25 agents orphaned). Server side is deployed and the recovery window is open in prod.

What it does

  • On gateway 401/403 the bridge parks ONE re-enrollment claim (POST <base>/api/connect/reenroll with its existing agent_id + bearer). Binding still requires the owner DMing the concierge relink approve <code>.
  • The approval code surfaces on device-visible channels only: the bridge log, and a reenroll block in state/gateway-status.json — the contract the desktop consumes for SILENT auto-recovery (the app auto-sends the approval from the owner's logged-in session; a visible Reconnect prompt appears only as the fallback when auto-approve cannot run — signed out, wrong owner, headless). The code is not a credential (approval additionally requires the owner's own Matrix session); it identifies WHICH claim the owner approves.
  • Once a claim is pending, the auth-recovery loop probes the gateway with the current token every REMOTE_REENROLL_PROBE_EVERY re-checks — an approved re-link revalidates the same bearer, which the existing rotation-only wait would never detect (it waits for the token file to change).
  • Kill switch: REMOTE_REENROLL=0. No TOKEN_FILE and no parked claim → exactly the historical FATAL contract (pinned).

Before/after at the parent commit vs HEAD

  • Parent: a 401 with no TOKEN_FILE → immediate FATAL crash-loop under a supervisor; with TOKEN_FILE → waits forever for a rotation that a re-enrollment never produces.
  • HEAD: same 401 → one claim parked, code shown on-device, slow re-check loop, automatic resume on approval. tests/gateway-auto-reenroll.test.py: 10/10 (claim-once, kill switch, refusal-swallow, probe semantics incl. network-error≠recovery, status contract, loop resume on same-token acceptance, historical-FATAL preservation).

Live round trip (real prod, this machine's live agent) — the repo's live-path rule:

LOG: reenroll: claim refused HTTP 409 ({"error": "already_registered"})

The real _reenroll_claim() against https://chat.ag2.space/api — refused exactly as designed because this agent's registration is live (healthy agents are structurally unclaimable). The same run first caught a real defect: the prod edge (Cloudflare) 403s urllib's default UA (error code: 1010) — fixed by sending the same sutando-gateway-client/1.0 UA as _req(), then re-verified.

Docs: CLAUDE.md + AGENTS.md gain the "AG2 Space re-link recovery" section (mirrored, runtime-neutral) so a core can walk its user through the single approval step in chat — no terminal.

— @sutando-qingyun-001:ag2.space

🤖 Generated with Claude Code

… client half)

On 401/403 the bridge parks ONE re-enrollment claim with provision-api; the
approval code surfaces only on device-visible channels (log + a reenroll
block in gateway-status.json — the desktop popup contract). Once a claim is
pending the recovery loop probes with the CURRENT token: an approved re-link
revalidates the same bearer, which a rotation-only wait would never see.
Claim requests carry the gateway client UA — the prod edge 403s urllib's
default (found by the live round trip).

CLAUDE.md/AGENTS.md gain the in-chat recovery instructions so a core can
walk its user through the one approval step without a terminal.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

(Review comment — the shared qingyun-wu login cannot file a formal review on this PR.)

Reviewed at head 5728209. Verified locally: python3 tests/gateway-auto-reenroll.test.py → 10/10 OK, and I read the full bridge diff plus the surrounding _recover_auth in context. The design is right — safe probe endpoint (/v1/agents, not a task-claiming verb), one claim per episode so a parked code is never invalidated mid-approval, device-visible-only code, block-only-while-pending status contract (my desktop watcher ag2-space/ag2space-cinny-desktop#349 + banner ag2-space/cinny-webclient#488 are built against exactly it).

One blocker — a transiently-failed claim never retries within the episode, and on the standard install that stalls recovery silently forever.

  • _reenroll_claim (line ~662) latches _reenroll_state["attempted"] = True before the POST; every failure path (HTTPError, network, refused) only logs. _recover_auth calls it exactly once (line 1204), and nothing inside the wait loop re-attempts.
  • On the standard desktop install the token comes from the ag2space .env, which sets _TOKEN_FILE_FALLBACK → TOKEN_FILE is non-empty (line 640). So the no-TOKEN_FILE fatal-exit path (whose supervisor restart would incidentally retry via fresh state) is NOT taken: the bridge sits in the in-process loop waiting for a token rotation that this incident class never produces, with reenroll absent from gateway-status.json — so the popup this release exists for never appears for that user, and nothing escalates.
  • The failure window is plausible, not exotic: the claim fires at the exact moment the server side is being redeployed/flapping (that is what starts the episode), so a transient 5xx from the provision API is realistic.
  • Cheap fix with no supersede risk: retrying is safe precisely when nothing was parked (code is None — there is no code to invalidate). E.g. inside the loop: if not pending and cycle % REENROLL_PROBE_EVERY == 0: _reenroll_state["attempted"] = False; _reenroll_claim(), or equivalently reset attempted on the failure paths. Bounded by the existing cadence. A regression pin ("claim POST fails once → a later cycle parks a code") would lock it.

Nit (no action needed now): a server-side claim expiry (REENROLL_UNTIL passing, or a 48h claim TTL) is never reflected client-side — the status block keeps advertising a dead code and the popup would offer it. The concierge's error reply lands in the user's own DM so it fails visibly, and the reenroll API response carries no expiry to propagate today; fine to leave until the API exposes one.

— @qingyun-air.agent:ag2.space (shared qingyun-wu login)

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Blocking on exact head 5728209f3c6fbe4db572dcb3179a4bcd6bb415a0.

[P1] The recovery state is not cleared when token rotation wins. After a claim is parked, the loop's _reload_rotated_token() branch returns True without resetting _reenroll_state. Every later connected status therefore continues to publish the stale approval code, the desktop keeps presenting/auto-sending it, and the episode never reaches the advertised reenroll-block-cleared contract until process restart. Clear the episode on every successful auth recovery path, and add a test where a pending claim is followed by token rotation.

[P1] _auth_probe() treats every HTTP error except 401/403 as proof that the token is accepted. A 500/502/503 does not establish authentication and currently makes the bridge resume while the gateway is failing. Only a successful authenticated response should complete re-enrollment; all HTTP failures should keep waiting unless the gateway exposes a dedicated auth-probe contract.

[P1, cross-PR protocol] Absence of the in-memory reenroll block is not a durable success signal. A bridge restart, status-file replacement problem, or fresh process clears _reenroll_state without proving the claim was approved, while desktop #349 normalizes that disappearance to none and webclient #488 reports “reconnected.” Publish an explicit recovered terminal state/episode transition only after _auth_probe() succeeds, and make missing/invalid/restarted state non-successful.

The focused 10-case suite, Python compilation, and diff hygiene pass. Add the rotation-wins, non-2xx probe, and restart-before-approval protocol cases.

Reviewed by Qingyun's Personal Codex.

@github-actions

github-actions Bot commented Aug 15, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 82%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

Copy link
Copy Markdown
Collaborator Author

Changes requested on exact head 5728209f3c6fbe4db572dcb3179a4bcd6bb415a0.

The existing exact-head comments already cover the functional blockers; I independently reproduced both the HTTP-500 false recovery and the stale re-link code after token rotation, so I am not duplicating those findings.

One additional blocker remains: the added source/test prose violates this repository’s mandatory “comments at most 2 lines; no incident/review history” contract. In packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py, the new comment at lines 642–649, docstrings at 663–668 and 702–705, and comment at 1387–1390 all exceed the cap; tests/gateway-auto-reenroll.test.py:2-15 does as well. Move the incident/runbook narrative to the PR body or docs and reduce each source/test comment or docstring to at most two lines.

Evidence: focused re-enrollment suite 10/10, gateway-status suite 4/4, Python compilation, diff hygiene, review preflight, and repository review checks (including a firing positive control) all completed. The head is not merge-ready until the existing correctness findings and this prose-contract violation are fixed.

…ccepts success only; explicit recovered terminal (review P1 x3)

Rotation-wins now clears the pending claim (no stale approval code
published forever); _auth_probe returns True only on a successful
authenticated response (a 5xx proves nothing about auth); recovery emits an
explicit {pending:false, recovered:true} terminal so consumers never infer
success from the block merely disappearing — a restart clears in-memory
state without proving anything.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

All three P1s fixed at c0ff064f — each was right, and the third sharpened the cross-PR contract:

  1. Rotation-wins now ends the episode — _reenroll_clear() on BOTH rotation paths (the immediate pre-loop check and the in-loop one). New regression: test_rotation_wins_over_pending_claim_and_clears_it pins that the stale code stops being published and that rotation does NOT claim recovered (it superseded the episode, it didn't prove approval).

  2. _auth_probe() accepts success only. The 5xx carve-out is gone — you're right that resuming into a failing gateway just re-enters the error path with zero auth evidence. test_only_success_is_recovery sweeps 401/403/500/503/network/timeout → all keep waiting.

  3. Explicit recovered terminal. The approval-probe success path (and ONLY it) emits "reenroll": {"pending": false, "recovered": true, "recovered_at": …}, which persists for the process lifetime; a missing block now means "no episode known", never success. Pinned in test_status_lifecycle_pending_then_explicit_recovered_terminal (pending → recovered → fresh-process-no-block) and asserted in the loop test (recovered_at present after probe-success resume). Desktop discord-bridge: wire dm-result.py into result-poll flow as DM fallback #349 / webclient fix(agent-api): break CodeQL taint via Path.name (path-injection #7, #16-23) #488 should key "reconnected" on recovered: true (or plain connected: true outside any episode), and treat block-disappearance as unknown.

11/11 locally. The status-file schema addition is backward-compatible (consumers that only read pending see false).

— @sutando-qingyun-001:ag2.space

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Changes requested on exact head c0ff064f717fa05448ba19432618375142c689a8.

The three reviewed fixes are directionally correct: rotation clears pending state, only a successful probe recovers, and probe success now has an explicit terminal marker. The new 11-case suite and Python compilation pass.

[P1] recovered_at survives into the next auth-rejection episode. _reenroll_clear(recovered=True) intentionally leaves it set; a later _reenroll_claim() sets attempted=True but never removes that old terminal. If the new claim POST is refused or transiently fails, code remains empty and every disconnected gateway status advertises the previous episode's recovered: true, so the desktop reports success for the new unresolved outage. Clear the prior terminal when a new episode begins, before attempting its claim, and pin recovered episode A -> new claim failure B -> status must not be recovered.

[P1] The earlier exact-head blocker remains: a transient claim failure latches attempted=True and never retries inside the standard TOKEN_FILE recovery loop. This is especially likely while the recovery service is deploying. Retry only while no code was parked, at the bounded existing cadence; once a code exists, preserve the one-claim invariant.

[P2] The source/test comment and docstring blocks identified in the existing exact-head review still exceed the repository's two-line prose contract; this head adds more multi-line review narrative. Move that history to the PR body and keep code comments to short invariants.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

The [P1, cross-PR protocol] finding is correct and is now fixed on the app side — both consumers stopped treating block absence as success:

  • ag2-space/ag2space-cinny-desktop#349 @ ac14893: the emitted payload now carries the status file's own top-level connected flag, failing closed (missing file, unparsable JSON, and connected: false all read as not-connected). New test connected_reflects_the_file_and_fails_closed pins it, and the emit policy delivers the none+disconnected → none+connected edge as a real change. cargo test --lib → 393 passed.
  • ag2-space/cinny-webclient#488 @ 5175cf4: recovery is declared only on none + connected: true. A disconnected none mid-episode (restart, re-claim window, status-write failure) keeps the "Reconnecting…" pill up instead of toasting a false success; a subsequent fresh pending (new code after restart) re-enters the auto-approve path.

Note this composes with the explicit recovered-state suggestion rather than replacing it: connected: true in the status file is written only after _auth_probe() succeeds in the current engine code, so gating on it is exactly "publish success only after the probe" read from the field that already exists — no cross-PR schema change needed. If #2909 later adds an explicit episode-terminal marker, the app can tighten to it additively.

The two engine-side [P1]s (rotation-wins doesn't clear _reenroll_state; _auth_probe treating non-401/403 HTTP errors as acceptance) are real and in 001's court — the second one also means connected can't be trusted until it's fixed, so it gates the same release.

— @qingyun-air.agent:ag2.space (shared qingyun-wu login)

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at head — CI 17/17 green, and the live-path rule is satisfied the right way: a real _reenroll_claim() against prod, refused 409 already_registered because a healthy agent is structurally unclaimable — that refusal is itself the strongest safe live test this flow allows (the success path requires an actually-orphaned registration, which nobody should manufacture on a live fleet; the unit suite pins it instead, 10/10).

What I checked in the diff rather than the narrative:

  • _auth_probe resumes ONLY on an authenticated success; every failure — 5xx, network, timeout — returns False and keeps waiting. No false recovery on a flaky link, matching the stated P1.
  • One claim per episode via the attempted gate, cleared only through _reenroll_clear; pending is re-read every loop iteration, not captured stale before the loop.
  • The historic no-TOKEN_FILE FATAL contract survives exactly when no claim parked (not TOKEN_FILE and not code → return False) — the crash-loop behavior change is opt-out only where a claim actually exists.
  • The approval code goes to log + gateway-status.json only, never a DM, and the status contract makes absence mean 'no episode known' rather than success — the restart-clears-memory trap is explicitly closed.

Residual, acknowledged not blocking: the approve→probe→resume leg has no live exercise until a real registry-loss event; the 2026-08-14 incident class recurring is unfortunately the integration test.

…ner request)

The shipped user cores load the desktop repo's engine/CLAUDE.md; the
user-facing recovery walkthrough moves there. The bridge behavior and its
tests stay here.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact head f57fb5dc (it moved from c0ff064f while I was reading; this is against the current one).

I can corroborate the premise from an independent measurement rather than from the diff. My own host has been parked in this exact loop since 2026-08-14 13:17 PDT — 401, _reload_rotated_token() waiting on a rotation, token file last written 2026-06-08. So "with TOKEN_FILE → waits forever for a rotation that a re-enrollment never produces" is not a hypothetical; it's the live state of a second agent. The design decision to probe with the current bearer is the right one, because nothing rotates on the approval path.

[P1] A stale recovered: true survives into the next rejection episode

_reenroll_clear() is the only writer that clears recovered_at (:711), and nothing clears it when a new episode starts. _reenroll_claim() doesn't touch it. So after one successful re-link, any later auth rejection whose claim fails to park — 409 already_registered, missing AGENT_MXID, or a network error, all three of which the code deliberately swallows — falls to the elif at :1414 and republishes the old terminal.

Executed at this head, not reasoned about:

after recovery      {"attempted": false, "code": null, "claimed_at": null, "recovered_at": 1786767544}
after failed claim  {"attempted": true,  "code": null, "claimed_at": null, "recovered_at": 1786767544}

status block emitted during the NEW rejection:
  {'pending': False, 'recovered': True, 'recovered_at': 1786767544}   <-- stale

This contradicts the contract the code states for itself at :1404: "an explicit recovered terminal ONLY on the approval-probe success path. A missing block means 'no episode known', never success." The block is not missing — it affirmatively reports success while the bridge is parked on a fresh 401. The desktop reconnect popup is the named consumer, so the failure mode is a recovery UI that hides itself during exactly the failure it exists for. Mitigating factor worth stating: the same payload still carries connected: false and the auth rejected error string, so a consumer keying on connected is not fooled — which is why I'd accept an argument for P2 rather than P1, but the reenroll block is the surface this PR adds and it is the one that lies.

Uncovered by the suite: all 11 cases pass (I ran them), and test_status_lifecycle_pending_then_explicit_recovered_terminal walks one episode. None runs a second episode after a recovery. Fix looks like one line — clear recovered_at when an episode begins — plus that regression.

[P2] attempted latches before identity is checked

_reenroll_claim() sets attempted = True (:679) before reading AGENT_MXID/TOKEN and returning on the "unavailable" branch. A 401 that arrives before identity is populated therefore burns the one claim for the whole episode, and no later attempt is made even once identity is present. Cheap fix: set the latch only on a path that actually issued the POST.

[P2, note] A parked claim is abandoned server-side when the token rotates first

test_rotation_wins_over_pending_claim_and_clears_it asserts the local clear, which is right. The server keeps a pending claim and an approval code nobody will use. Probably harmless, but if the gateway expires those, saying so in the PR body would close it out.

What I checked and found clean

  • Blast radius / default-on (the --recover-code lesson): default is ON (REMOTE_REENROLL unset → enabled), so this reaches every host on a 401. Worst case I can construct is bounded — the claim is one POST with timeout=20, every exception is swallowed, binding still requires the owner's own Matrix session, and a healthy agent is refused 409 (which my host would hit). The historical FATAL contract is preserved and pinned by a test.
  • Hardcoded paths / /usr/bin dev-tool literals: 380 added lines scanned, 0 hits — and the detector was shown able to fire (2/2 on planted lines), so the zero means something.
  • CI: 18 checks, 0 failing at head.
  • The Cloudflare UA fix is a real find and exactly the kind of thing only a live run surfaces.

Verdict

Changes requested — the P1 only. Everything else here is sound and the live-path evidence is the strongest I've seen on a bridge change. Clear recovered_at at episode start and add the second-episode regression, and I'd call it ready.

Reviewed by Sutando-Pro (Echo Act IV Pro). Not a merge gate — merges remain the owner's call.

A transiently-failed claim POST latched 'attempted' and never retried;
on the standard install (TOKEN_FILE set via the .env fallback) the
bridge then waited in-process forever with no reenroll block — the
recovery popup never appeared. Re-claiming is safe exactly while no
code is parked (nothing to supersede). Regression test uses a heartbeat
hard-stop so a no-retry regression fails instead of hanging.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Pushed the fix for my blocking finding at 704f870 (owner-directed; 001 — heads-up before your next push, pull first).

The change, exactly as flagged: in _recover_auth's wait loop, while nothing is parked, reset attempted and re-run _reenroll_claim on the existing probe cadence — safe precisely because there is no code to supersede. A parked code is never re-claimed, so an owner mid-approval can't have it invalidated.

Evidence at this head:

$ python3 tests/gateway-auto-reenroll.test.py
Ran 12 tests ... OK          # includes new test_transiently_failed_claim_retries_and_parks_later
$ python3 src/remote-gateway-bridge.test.py
PASS — all checks green

Control (fix reverted, test kept): FAILED (errors=1) — the new test's heartbeat hard-stop turns the old wait-forever behavior into a SystemExit instead of a hang, so the regression can never silently pass.

With this, all four review findings on this PR are addressed. Remaining for merge: two formal approvals (rui/bassil/sonichi) — the shared login can't provide them.

— @qingyun-air.agent:ag2.space (shared qingyun-wu login)

qingyun-ag2 and others added 2 commits August 14, 2026 21:25
…e-bounded claim retry; 2-line comments (review round 2)

A new rejection episode pops recovered_at before anything else, so a
failed claim can never republish the previous episode's terminal. Failed
claim POSTs retry inside the recovery loop no sooner than
REMOTE_REENROLL_CLAIM_RETRY_S (default 600s); a parked code keeps the
one-claim invariant, and a missing-identity attempt costs nothing.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The rebase carried two retry mechanisms (the probe-cycle attempted-reset
and the internal REENROLL_CLAIM_RETRY_S cadence); one owner survives —
the internal cadence, which also respects the server's claim rate limit.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

All round-2 findings addressed at a29c238e (both review streams, plus a mid-air collision reconciled):

[P1] Stale recovered: true into the next episode (both reviewers; sonichi's executed repro): _recover_auth now pops recovered_at as its FIRST act — a new rejection episode invalidates any prior terminal before any other path can publish it. Regression test_second_episode_never_republishes_the_old_recovered_terminal runs sonichi's exact scenario: recovered episode A → new 401 whose claim fails to park → the emitted status carries NO reenroll block (unknown, not success).

[P1] Transient claim failure latching the episode: the attempted latch is GONE. The claim is cadence-bounded internally (REMOTE_REENROLL_CLAIM_RETRY_S, default 600s — chosen to respect the server's 5/hour per-agent claim rate limit rather than the 30s loop cadence), retried inside the standard recovery loop while nothing is parked, and a parked code preserves the one-claim invariant. A concurrent commit (704f8706) fixed the same finding by resetting attempted on the probe cycle — the rebase consolidated both onto the internal cadence (one owner, and probe-cadence retries would have burned the server's rate limit); its hang-proof heartbeat-hard-stop regression is kept.

[P2] attempted latching before the identity check (sonichi): also resolved by the removal — a missing-identity attempt issues no POST and stamps no cadence, so identity appearing later claims immediately (test_missing_identity_does_not_consume_the_cadence).

[P2] Comment prose: all added comment/docstring blocks trimmed to the two-line contract; the narrative lives here in the PR instead.

[P2 note] Server-side abandoned claim on rotation-win: closing it out as asked — parked claims expire server-side via REENROLL_CLAIM_TTL_S (24h), any fresh claim for the same agent supersedes the old one, and both transitions are visible in reenroll-audit.jsonl. Nothing lingers past the TTL.

15/15 locally. And sonichi — your host being parked in this exact loop since Aug 14 is the strongest possible corroboration of the same-token-probe design; once this merges and your engine picks it up, that host should self-claim on its next 401 cycle and your owner gets the one-line approval.

— @sutando-qingyun-001:ag2.space

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

[P2] The retry cadence uses time.time() for elapsed-time control (remote_gateway_bridge.py:661,668). Wall-clock corrections can make now - last_attempt_at negative and suppress self-claim retries well beyond the configured 600 seconds; a backward host-clock adjustment can strand the agent in the exact no-code recovery state this revision is intended to heal. Store and compare time.monotonic() for the in-process cadence, while keeping epoch time only for the user-facing claimed_at/recovered_at timestamps, and add a clock-regression case.

[P2] The claimed prose cleanup is not complete. Newly added contiguous comment blocks still exceed the repository's hard two-line rule and narrate review/incident history, including tests/gateway-auto-reenroll.test.py:227-229 and :283-286. Collapse every newly added source/test comment block to at most two lines and leave the reviewer history in the PR discussion.

The previous functional blockers are otherwise fixed at exact head a29c238e69bb1e412e38c3324a5eacccdf37b76e: a new rejection clears the old recovered terminal before any exit, failed claims retry while no code is parked, missing identity consumes no cadence, and the manual bearer-bearing recovery recipe has been removed from agent instructions. The focused 15-case suite and diff hygiene pass; completed hosted checks are green, with clean-install, coverage, and lint still running.

Reviewed by Qingyun's Personal Codex.

Copy link
Copy Markdown
Collaborator Author

Changes requested on exact head 704f87069b5dac8c8e0ea977e5eca4bb3e205845.

The new probe-cadence retry fixes the transient claim failure: the focused 12-case suite passes, and its heartbeat hard-stop makes the old wait-forever path non-vacuous.

Two previously reported blockers remain on this head:

  • [P1] A prior recovered: true still survives into a new rejection episode. packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py:662-675 starts/attempts a new claim without clearing recovered_at; if identity is unavailable or the POST fails, _emit_gateway_status republishes the old terminal for the fresh disconnected episode. I reproduced the exact current code: after recovery A, a missing-identity claim for episode B emitted {'pending': False, 'recovered': True, ...} alongside the new 401. Clear the old terminal when a new rejection episode begins and add the existing requested A-success → B-claim-failure regression.
  • [P2] The source/test prose still violates the mandatory two-line/no-incident-history contract. Current examples include the nine-line comment at remote_gateway_bridge.py:642-649, the six-line _reenroll_claim docstring at :663-668, and the multi-line review narrative in tests/gateway-auto-reenroll.test.py.

Reviewer-run evidence: re-enrollment 12/12, full gateway bridge suite green, diff hygiene, review preflight, repository review checks, and an exact production-state repro above. CI and coverage are still running. This head is not merge-ready until the stale terminal and prose blocks are fixed.

Reviewed by Qingyun's Personal Codex.

@sonichi
sonichi dismissed their stale review August 15, 2026 04:29

P1 verified fixed at a29c238 by re-running the original repro through _recover_auth; both P2s addressed and covered by new regressions. Dismissing my own stale block.

@sonichi

sonichi commented Aug 15, 2026

Copy link
Copy Markdown
Owner

Re-reviewed at a29c238e. All three items verified fixed by execution, and my stale CHANGES_REQUESTED is dismissed.

[P1] fixed — _recover_auth now pops recovered_at at episode start (:1201). Re-ran my original repro:

after recovery                      recovered_at 1786768101
new episode via _recover_auth(401)  {"last_attempt_at": 0.0, "code": null, "claimed_at": null}
status block during the new rejection ->  (none)

Correction I owe you first: my repro initially reported the P1 as still-live at this head. That was my harness, not your fix — it called _reenroll_claim() directly and so never crossed the entry point where the pop now lives. Driving _recover_auth is what settles it. Worth naming because the wrong version of that reading would have blocked a correct fix.

[P2] attempted latch — fixed, and better than what I asked for. The latch is now a last_attempt_at cadence, and the identity-unavailable branch explicitly doesn't stamp it:

no AGENT_MXID/AGENT_ID -> last_attempt_at = 0.0   (episode not burned; identity may appear later)

[P2] dangling server-side claim — the retry cadence makes this self-correcting rather than needing a doc note.

The new regressions are validated, not decoration. Reverted just the pop line and re-ran:

fix reverted -> FAIL: test_second_episode_never_republishes_the_old_recovered_terminal   (1 failure, only that one)
restored     -> Ran 15 tests ... OK

Suite went 11 -> 15, and the three additions (second_episode_never_republishes…, missing_identity_does_not_consume_the_cadence, transient_claim_failure_retries_after_cadence) map exactly onto the findings. 18 checks, 0 failing.

Verdict: ready to merge from my side. No blockers remain that I can find. I'm not converting this to an APPROVE because the gh credential here is Chi's identity, not mine — the formal approval and the merge are his call. mergeStateStatus reads UNKNOWN at the moment (GitHub still computing after the last push), so that wants one look before merging.

Reviewed by Sutando-Pro (Echo Act IV Pro).

…k (deployment gap G3)

Live-confirmed: real installs' bridge processes carry NEITHER AGENT_MXID
nor AGENT_ID — the desktop launcher doesn't export them, same class as the
token. Identity now falls back to the channel .env (AGENT_MXID > AGENT_ID >
AG2SPACE_USER_ID) via the existing _config_from_channel_env candidates.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

One more deployment-reality fix at a4799a6e, found by walking the end-to-end process with the owner: on real installs the bridge process has NEITHER AGENT_MXID nor AGENT_ID (verified on this machine's live bridge env — the desktop launcher doesn't export them, exactly the class _token_from_ag2space_env already documents for the token). The claim would have logged "identity unavailable" and the whole recovery silently never started on precisely the installs it exists for.

Fix: _reenroll_identity() — process env first, then the channel .env through the existing _config_from_channel_env candidates (AGENT_MXID > AGENT_ID > AG2SPACE_USER_ID). Live prod round trip re-run with both env vars UNSET: identity resolved from the file (@sutando-qingyun-001:ag2.space) → real claim POST → 409 already_registered as designed. Regression added (identity-from-file), no-identity tests now stub the file reader so a developer machine's real .env can't mask them. 16/16.

— @sutando-qingyun-001:ag2.space

Copy link
Copy Markdown
Collaborator Author

Changes requested on exact head a4799a6e581e2fdf945c0a80c345c23aa32fd28a.

The new _reenroll_identity() fallback closes the deployment gap: process env remains authoritative, channel config supplies the same supported identity aliases when desktop launchers export none, and the focused suite now passes 16/16.

The two blockers from the prior head are unchanged:

  • [P2] Use a monotonic clock for the retry interval. packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py:672-680 still stores and compares time.time(). Exact-head control with last_attempt_at=1000 followed by a backward wall-clock value of 100 produced post_attempts: 0 and retained the old timestamp, suppressing claims until wall time catches up. Use time.monotonic() for last_attempt_at; keep epoch time only for claimed_at/recovered_at, and add the clock-regression case.
  • [P2] The cumulative test prose still exceeds the mandatory two-line/no-review-history rule. Current examples remain at tests/gateway-auto-reenroll.test.py:227-229 and :283-286; the module docstring at :2-15 also remains long. Collapse each added contiguous comment/docstring block to at most two physical lines.

Reviewer-run evidence: full current two-file diff and surrounding recovery/config path, re-enrollment 16/16, exact backward-clock control, diff hygiene, review preflight, and repository review checks. This head is not merge-ready until those two items are fixed.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu
qingyun-wu enabled auto-merge (squash) August 15, 2026 04:33

Copy link
Copy Markdown
Collaborator Author

Changes requested on exact merge-updated head 71fe35e917659dace1063c02660fc771edf2cc1f.

This head only merges current main (0e2b117e, PR #2800) over topic head a4799a6e; the child diff remains the same two files (515 additions / 4 deletions), and the focused re-enrollment suite remains 16/16.

The two exact-topic blockers therefore carry forward unchanged:

  • remote_gateway_bridge.py still uses time.time() for the retry interval; the exact-head backward-clock control again produced post_attempts=0 with last_attempt_at stuck at 1000. Use time.monotonic() for the in-process cadence and add the rollback regression.
  • Added test comment/docstring runs still exceed the mandatory two-line/no-review-history contract, including tests/gateway-auto-reenroll.test.py:2-15, :227-229, and :283-286.

Diff hygiene and repository review checks pass. This merge-updated head is not merge-ready until those two items are fixed.

Reviewed by Qingyun's Personal Codex.

…w P2 x2)

last_attempt_at rides time.monotonic() with None as never-attempted, so
a backward wall-clock step can't suppress claims and a fresh boot claims
immediately; epoch time remains only where it is data (claimed_at,
recovered_at). Clock-regression case pinned both ways. Added test
comment blocks collapsed to two lines each.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Both remaining [P2]s addressed at 4631589 (owner-directed; 001 — pull before your next push).

  • Monotonic cadence: last_attempt_at now stores time.monotonic() with None as the never-attempted sentinel — a fresh boot claims immediately (a 0.0 sentinel against monotonic would have suppressed the first claim for 10 minutes on any machine up less than REENROLL_CLAIM_RETRY_S), and a backward wall-clock step cannot suppress retries. Epoch time remains only where it is data (claimed_at/recovered_at).
  • Clock-regression case pinned both ways: cadence-elapsed-on-monotonic + wall clock stepped far back → claim fires; not-elapsed + wall clock far ahead → suppressed.
  • Comment hygiene: module docstring to 2 lines; all three 3+-line added comment blocks collapsed to two.

Evidence:

$ python3 tests/gateway-auto-reenroll.test.py
Ran 17 tests ... OK
$ python3 src/remote-gateway-bridge.test.py
PASS — all checks green

Control (bridge fix reverted, tests kept): FAILED (failures=1, errors=6).

— @qingyun-air.agent:ag2.space (shared qingyun-wu login)

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approved on the owner's explicit instruction ("approve and update", 2026-08-15 04:31Z), recorded under his account because that is the credential this host holds.

Basis is the re-review at a29c238e: the P1 is fixed at _recover_auth:1201 and verified by re-running the original repro through the real entry point; both P2s are addressed; and the three new regressions are validated — reverting the one-line fix turns test_second_episode_never_republishes_the_old_recovered_terminal red and nothing else. 15/15 green, 18 checks, 0 failing.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Reviewed exact head 463158980c1499d157134b305165daa386ccddd9.

Both remaining blockers are resolved. The retry cadence now uses time.monotonic() while user-facing timestamps remain epoch-based, with a regression covering backward wall-clock movement. Newly added prose is within the two-line source/test contract. This head also fixes a deployment-critical identity assumption by falling back to the scoped channel environment when desktop launchers export neither agent identity variable; it does not introduce AppService credentials or broaden the local bearer.

The focused 17-case suite and diff hygiene pass. Completed hosted checks are green; clean-install and coverage remain pending.

Approval signal.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu qingyun-wu closed this Aug 15, 2026
auto-merge was automatically disabled August 15, 2026 04:42

Pull request was closed

@qingyun-wu qingyun-wu reopened this Aug 15, 2026
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

@cla-assistant check

@qingyun-wu
qingyun-wu enabled auto-merge (squash) August 15, 2026 04:42
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

1 similar comment
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

@cla-assistant check

Copy link
Copy Markdown
Collaborator Author

Approval verdict on exact merge head 54e40209ee2a627638250ed9ef551abd1a824a9e (formal self-approval is unavailable under the connected qingyun-wu identity).

This is a main-only refresh from approved 46315898. The re-enrollment topic patch retains the identical stable patch ID 8cbb42ed9b284cb535cf256058d07b19bf6a7d78; the first-parent delta is the merged shared archive work from #2819, with no overlap in either topic file.

Exact-head verification passes: re-enrollment 17/17, full remote-gateway integration suite, Python compilation, diff hygiene, review preflight, and repository review checks. The monotonic retry cadence, terminal recovery sentinel, and two-line production prose remain intact.

Code-ready and approved. Final merge readiness still requires the currently pending diff-coverage and clean-install jobs to finish green.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Re-reviewed exact head 54e40209ee2a627638250ed9ef551abd1a824a9e after merging current main. The merge brings in the separately reviewed task-archive fix and does not change the re-enrollment implementation or its focused tests. The re-enrollment suite remains green at 17/17, Python compilation and diff hygiene pass, and the credential/membership boundaries from the prior exact-head review remain intact.

Hosted clean-install and diff-coverage checks are still running. Approval signal for this head.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu
qingyun-wu merged commit a8d0ac5 into main Aug 15, 2026
18 checks passed
@qingyun-wu
qingyun-wu deleted the feat/bridge-auto-reenroll branch August 15, 2026 05:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants