Repository navigation
feat(gateway): task lease generation, per-task beats, cancel and the turn gate [AgentActivity] - #5883
Merged
marklysze merged 35 commits intoOct 30, 2026
Merged
Conversation
Adds src/agent_turn.py (vendored into ag2-sparrow): the 0400 turn file at state/turns/<task_id>_g<gen>_<ulid>.json, its .closed fence, the journal reader for dispatch_intent/dispatch_outcome records, and classify_turn. A plain-text answer may be sent only when the turn has no bound intent, no unbound intent since it began, and core-api confirms zero operations.
…gate on results Behind REMOTE_TASK_LIFECYCLE_V2 (default off) the bridge sends one lease-holder worker_id and protocol_version 2 on ack, records the execution_generation the ack reports, beats each live attempt on POST /v1/tasks/heartbeat, and puts the generation and worker_id on every result. A fenced attempt (task_cancelled, stale_execution_generation) stops the core through a CANCEL_INSTRUCTION task and is cancel-acked; its late result is settled, never posted. Every five minutes GET /v1/tasks/active is reconciled with the in-flight set. With REMOTE_TASK_TURN_CONTEXT too, each attempt's turn file is written on ack and closed when its result settles, and the result is gated on the turn's journal: skipped when the task closed terminally through MCP, posted as waiting_input + no_send while suspended, and posted with no_send when the same text was already sent as a room message. A result 409 that ends the task, or a 400 waiting_input_without_blocker, settles the item instead of quarantining it. A re-served task carrying `wake` is left unacked rather than closed as a redelivery.
…arison apart from the delivery reconcile
Contributor
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 94%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.pyLines 3985-3994 3985 except FileNotFoundError:
3986 _ATTEMPTS._update(wire_tid, taken_ctl=ctl)
3987 _log(f"task {wire_tid}: cancel {ctl} for generation {old} was already taken by the core — "
3988 f"generation {gen} may be stopped")
! 3989 except OSError as e:
! 3990 _log(f"task {wire_tid}: cancel {ctl} could not be withdrawn ({diagnostics.describe(e)})")
3991
3992
3993 def _ack_fenced(tid: str, lease_id: "str | None", raw: str, published: bool) -> bool:
3994 """An ack refused with a fencing code: the attempt is fenced, its pending ack retired and,Lines 4068-4078 4068 break # the core's turn has touched it: taken
4069 path.unlink()
4070 except FileNotFoundError:
4071 continue
! 4072 except OSError as e:
! 4073 _log(f"task {tid}: {why}, its task file could not be withdrawn ({diagnostics.describe(e)})")
! 4074 break
4075 _log(f"task {tid}: {why} — its untaken task file was withdrawn")
4076 return
4077 _queue_cancel_instruction(tid, why)
4078 _log(f"task {tid}: {why} — the core has taken it up and is asked to stop")Lines 8192-8201 8192 continue
8193 if headers.get("source") == task_attempts.CONTROL_SOURCE:
8194 try:
8195 _retire_orphan(rfile, ARCHIVE_RESULTS_DIR, tid, identity_of(rfile)[1])
! 8196 except OSError:
! 8197 pass
8198 continue
8199 # One open decides every arm below, and only that generation is retired.
8200 try:
8201 data, generation = identity_of(rfile)packages/ag2-sparrow/ag2_sparrow/task_attempts.pyLines 284-292 284
285 def beating(self, wire_tids) -> bool:
286 """Whether any of these attempts is due per-task beats."""
287 if self._task_beat_disabled:
! 288 return False
289 data = self._load()
290 return any(_beats(data.get(t) or {}) for t in wire_tids)
291
292 def fence(self, wire_tid: str, error: dict) -> dict:Lines 400-408 400 return None
401 if not rec.get("turn_id"):
402 # No turn: only the records naming this task since it was published can rule.
403 if rec.get("published_at") is None:
! 404 return None
405 records = agent_turn.read_turn_records(self.state_dir, None, rec["published_at"],
406 task_id=wire_tid)
407 if not (records.intents or records.outcomes or records.partial or not records.readable):
408 return NoneLines 508-517 508 return None
509 finally:
510 try:
511 os.unlink(tmp)
! 512 except OSError:
! 513 pass
514
515
516 def cancel_task_text(local_tid: str, task_id: str, timestamp: str, reason: str) -> str:
517 """The core's existing cancel signal, from a source the bridge never posts back.""" |
…ask names The coverage gate measures tests/**/*.test.py in process: the package test moves to tests/sparrow-task-attempts.test.py, the bridge hooks get an in-process suite, and the vendored agent_turn.py copy is omitted like the other src/ copies. Two cancels in one millisecond no longer share a control-task name, and a recorded acked_at of 0 is read as itself.
…erminal nor suspended
…urn-gate # Conflicts: # packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py
…re, a corrupt journal is unreadable Under REMOTE_TASK_LIFECYCLE_V2 a result written before its task's ack named the lease waits for that ack (at most 120 s), and records the proxy could not bind but that name the task count as the attempt's turn from publication on, so neither the lease fields nor the journal gate can be skipped by a fast core. Every fencing error_code on either ack path (404 or 409) fences the attempt, retires its pending ack and, for a published task, stops the core, except a stale older lease whose newer serve is pending here. A journal line that does not parse makes the journal unreadable instead of absent; an unterminated last line younger than 30 s defers the result. A served task carrying `wake` is left unacked whether or not this node saw its first attempt. A late ack for a task no longer queued opens no turn.
marklysze
added a commit
to marklysze/sutando
that referenced
this pull request
Oct 25, 2026
…pt to settle and always runs with its answer Merges feat/agent-activity-turn-gate. A served `wake` is decided before any local shortcut: run with its answer under turn context, or left unacked (turn context off, ack_first, an episode already run, or no usable episode), whether or not this node saw the first attempt. A wake served while an earlier attempt of the same id is still settling here is left unacked until it is served again, under a lock the settle also holds, so that settle can never retire the wake. Finishing an attempt checks it is still the one the result belonged to, and opening a wake starts a fresh turn.
…edelivery keeps the journal boundary On a broker that has ever named a lease, a result whose ack never answered keeps waiting until the ack lands, the attempt is fenced, or the server closes the task. On a broker never seen to name one, the 120 s fallback is gated on the journal records that name the task, so an early send, terminal action or blocker still rules. A task served again before its ack keeps its first publication time, so its earlier unbound room actions stay inside the turn's journal window.
…r reconcile from its first publication A result held because its ack never named the lease on a broker that names leases is now released by the reconcile too: once the complete active listing omits the task past the grace, the attempt is fenced, the core stopped, and the result settled without a post.
… every held task beats A task_cancelled in a per-task beat no longer fences the attempt. The core gets one CANCEL_INSTRUCTION task; once it has handled it, a result it wrote for the task is posted normally with no cancel-ack, and with no result the stop is confirmed on cancel-ack and a later result is settled. A task_cancelled on the ack still means the task is not started and is cancel-acked at once. The per-task beat now runs on its own 60 s cadence ahead of the client heartbeat's backoff, so a Stop reaches every task the bridge holds for as long as it holds it.
… stopped, not no_action_needed While a Stop is pending, a core result that is no_send or empty confirms the Stop with a cancel-ack and is settled locally; it is never posted as no_action_needed. A failed cancel-ack holds the result for the next pass. A result with a body is still posted (a finished reply is never dropped).
… ~15 s TASK_HEARTBEAT_INTERVAL_S (15) replaces the 60 s client cadence for per-task beats; the client heartbeat stays at 60 s. While an attempt with a generation is held, the task long poll returns when the next beat is due, so the beat is not delayed by a 25 s poll. Attempts without a generation still never beat.
…urn-gate # Conflicts: # packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py
…ed after 10 min, not beaten forever On every per-task beat, an attempt with a generation whose task file is no longer in the queue (archived or removed), with no result file, no claim and no turn activity from the session's own hooks for DISPATCH_LOST_S (600 s), is marked dispatch_lost: its beats stop and a [no-send] result file is created only if none exists (O_EXCL, so a real answer always wins). The result posts with status failed and no_send. A file put back in the queue resets the clock.
…urn-gate # Conflicts: # CHANGELOG-PENDING.md
…after 30 min A turn that ends without a result leaves its task file queued, so the per-task beat renewed the lease forever and the room showed the task as working. Once a turn row newer than the file exists and nothing more is written for 30 min, the attempt stops beating and posts failed + no_send. A file the core has not reached yet keeps beating.
…d from flight The in-flight reconcile dropped an id whose task file was archived without a result as "completed elsewhere", even while its attempt still beat. Its lease then lapsed, the broker re-served it, and the cycle repeated with nothing posted. A beating attempt is now left to the 10-min dispatch watch, which posts it as failed with no_send.
…journal lock close_turn takes <state>/room-actions.jsonl.lock with the proxy's format (O_EXCL, "<pid> <token> <start_ms>"), breaks it only for a dead pid and waits at most 2 s for a live holder, so a close never lands between the proxy's closed-check and its intent append. Past the wait the turn is still fenced, and the bridge logs it. Only turn-context attempts have turns to close.
sonichi
reviewed
Oct 30, 2026
sonichi
left a comment
sonichi
left a comment
Owner
There was a problem hiding this comment.
Reviewed this layer against main at 2d8dd86. Not approving yet: two of these concerns need an answer first. CI is green, and agent-turn, sparrow-task-attempts, task-dispatch-contract and turn-ledger-stop-gate pass locally.
- The reconcile can drop a finished reply.
_reconcile_ledger/task_attempts.py~461-472: a held attempt older than 120 s that/v1/tasks/active?status=outstandingdoesn't list is fenced astask_already_terminal, and its result is then settled without being posted. If "outstanding" leaves out leased or working tasks, or lags behind, a reply the core is still writing is lost, and only a log line records it. Please state the backend contract for "outstanding", or only fence on an explicit terminal status. - One bad journal line fails every task. In
agent_turn.py~296-307,state/room-actions.jsonlis shared by every task. One unparseable line older than 30 s (a torn append) makes the journal unreadable, so every result under turn context postsfailed+no_senduntil the journal rotates twice. A bad line should be skipped, or fail only the task it belongs to. - A reply can be posted twice across a generation change. After a re-serve as generation 2, the gate reads only generation 2's records, so a reply already sent under generation 1 is sent again. The gate should also count the same task's earlier generations as "already replied".
STALLED_S = 1800can close a long turn. If the core writes no hook rows for 30 min (one long tool call, or perhaps a foreground subagent wait), the bridge posts a[no-send]failure, and the real answer arrives afterwards.
Changes that apply even with the flags off (please list them in the body, or put them behind the flag):
/v1/results409task_already_terminal | stale_execution_generation | task_cancelledand 400waiting_input_without_blockernow count as delivered;- any task carrying
wakeis left unacked; - the orphan sweep retires results from
source: relay-controltasks.
The second matters once the backend sends wakes: an agent can still call room.question.ask with the flags off, and its task would then sit unacked for 24 h.
Stale-generation fencing within one install looks sound (_withdraw_stale_cancel, generation read at post time).
(agent-assisted, Blue for Chi's seat)
keweichen
requested changes
Oct 30, 2026
keweichen
left a comment
keweichen
left a comment
Collaborator
There was a problem hiding this comment.
Changes requested at 2d8dd86f54d7e5a5851a7f698eb69dfabf29a394 — not ready to merge.
Blocker 1 — the journal-lock timeout breaks the serialization guarantee. src/agent_turn.py:247-263 calls _take_journal_lock(), but when it returns None after two seconds, close_turn() still writes .closed and removes the turn JSON. Exact-head repro while a live proxy critical section held the lock:
close_returned_lock=False
proxy_lock_still_held=True
closed_marker_landed_inside_proxy_section=True
turn_file_removed=True
That allows the close to land after the proxy's closed check but before its intent append, directly contradicting docs/remote-gateway-protocol.md:425-427 and reopening the duplicate-send race. Closing/removing must require the lock; failure needs a durable retry path.
Blocker 2 — timeout alone can auto-fail valid work. task_attempts.py:267-272 declares a task stalled from its last activity row plus 30 minutes, and _watch_dispatch() at remote_gateway_bridge.py:4129-4142 has no process/core-liveness input. A legitimate tool call longer than 30 minutes emits its working row before the call and then reproduces stalled_due_for_still_running_30m_tool=True; the bridge posts failure and can reject the eventual real answer. Default-off limits today's blast radius, but this is destructive when rollout enables it.
Blocker 3 — a transient result write failure permanently suppresses retry. _write_if_absent() at remote_gateway_bridge.py:678-696 returns True after os.write/fsync failure merely because its empty placeholder exists. _fail_unanswered() leaves dispatch_lost set, _beats() stops heartbeats, and result_ready never drains the zero-byte file. Exact repro: write_if_absent_return=True, placeholder_size=0, result_ready=False, beats_after_dispatch_lost=False. The test at tests/gateway-agent-activity-hooks.test.py:685-688 currently codifies the unsafe result. Remove the failed placeholder safely and retry.
Blocker 4 — the exact live layer lacks a post-restart witness. The named deployed engine is not this head, and the body explicitly says the 15-second cadence was not run live. This startup/network/delivery path needs a real current-layer beat/fence round trip per REVIEW.md lesson 15.
Mandatory preflight, focused exact-head suites, vendor sync, scanner with positive control, CI, and CLA otherwise pass. Worst case with the opt-in enabled is duplicate delivery or a valid task being failed and losing its answer.
…orn lines, generations, stalls, locked closes - Reconcile: not being listed as outstanding no longer fences an attempt; only a beat's or ack's explicit terminal answer does. The active list is read as one page of 200; a longer listing skips the pass instead of walking a cursor. - A torn journal line fails only the turn or task it names; another task's torn line is skipped. - A reply sent as a room message under an earlier generation of the same task counts as already sent. - Stall: never while the core's health row reads moving (mid-turn); 30 min when it reads idle; 2 h when its state is unknown. - close_turn writes the .closed marker only while holding the proxy's journal lock; otherwise nothing changes and close_finished_turns retries it on the beat cadence, from the still-open turn file. - A failure result is written to a temp file and linked into place, so a failed write leaves nothing behind and is retried. - A task carrying `wake` is left unacked only with REMOTE_TASK_LIFECYCLE_V2=1; with the flag off it is handled as on main.
The bridge may not hard-link (its links are reserved for result disposal), so create-if-absent moves to task_attempts.write_if_absent; the bridge keeps only its logging. Also trims the stall comment to two lines.
sonichi
requested changes
Oct 30, 2026
sonichi
left a comment
sonichi
left a comment
Owner
There was a problem hiding this comment.
Re-reviewed the delta 2d8dd86...6dabc1e (69925d8, 6dabc1e). Agent-assisted, Blue for Chi's seat. Three of my four items are fixed and one is mostly fixed; one new crash needs fixing first.
Earlier items
- Fixed.
compare_with_serverreturnsnot_listedand fences nothing. Only an explicit terminal status from a beat or ack fences (_fence_attempt), and the protocol doc says so. Skipping the reconcile when the listing has more than one page is the safer choice. - Fixed, but with a crash (A below). A bad line now fails only the turn or task it names.
- Mostly fixed. An earlier generation's room-message send now counts as already replied (see B).
- Fixed. There's no stall while the core reads as mid-turn: 30 min when idle, 2 h when unknown.
New
- A (blocking).
read_turn_recordsraisesTypeErrorwhenturn_idis None.- Location:
src/agent_turn.py:315, and the vendored copy. - Cause:
elif turn_id in line or …runs for an attempt with no turn, andresult_decisionpasses None (task_attempts.py:405). - Probe: with any stale unparseable line in the shared journal, a direct call raises
'in <string>' requires string as left operand, not NoneType. With turn context on,_deliver_result_payloadthen fails on every pass until the journal rotates. - Fix:
(turn_id and turn_id in line), plus a test of the no-turn branch with a stale bad line.
- Location:
- B. An earlier generation's send intent suppresses the new generation's reply even if that send failed (
agent_turn.py~327-329, 454). Generation 1 logs the intent, the send errors or the worker dies, and generation 2's answer is postedcompleted, no_send, so the user gets nothing. Count a sibling only when it has a successful outcome (or no failed one). A generation change is exactly the crash-and-reclaim case this has to handle. - C. Docs are stale. The wake-unacked behaviour is now gated on
REMOTE_TASK_LIFECYCLE_V2(bridge ~4809), which is right. But the PR body still lists it under "Always on", anddocs/remote-gateway-protocol.md~420 states it with no flag.
Smaller notes, not blocking
- A line torn before its ids now fails open for every task. That's usually right, since nothing was sent, but worth a line in the docstring.
task_attempts.write_if_absentis a third copy of the link-to-publish helper (local_record.create_text_whole,agent_turn.write_turn_context). It doesn't fsync the directory afterlink, and it can leave a temp dotfile on a crash.- Until a deferred
_close_turnlands, the turn isn't fenced. That's the intended trade-off, but it should be stated._retry_turn_closescan also block the main loop for up to 2 s per open turn while the proxy holds the journal lock. _core_motionignoresalive/condition, so a wedged core that still reads "moving" never stalls.
Tests: agent-turn (39), sparrow-task-attempts (28) and gateway-agent-activity-hooks (54) pass locally on a fresh clone at 6dabc1e, plus the probe for A. CI was partly pending.
Collaborator
Author
|
Addressed the review points from both reviews on 2d8dd86, in Reconcile cannot drop a finished reply (sonichi 1)
One torn journal line (sonichi 2)
Double post across generations (sonichi 3)
Stall can close a long turn (sonichi 4, keweichen 2)
Journal-lock timeout (keweichen 1)
Failed result write (keweichen 3)
Flag-off behaviour
Live witness (keweichen 4): a current-layer beat/fence round trip is being scheduled on the rig. It will be pasted here when run. Suites: |
sonichi
reviewed
Oct 30, 2026
sonichi
left a comment
sonichi
left a comment
Owner
There was a problem hiding this comment.
Still open at 573004d (agent-assisted, Blue for Chi's seat). The push since my request-changes is a main merge only, and both items reproduce:
src/agent_turn.py:315still runsturn_id in linewithturn_id=None(fromtask_attempts.result_decision). Probe: one stale torn line plus a no-turn attempt raisesTypeError: 'in <string>' requires string as left operand, not NoneType.agent_turn.py:327still counts an earlier generation'sroom.message.sendintent even when its outcome failed. Probe: generation 1's send failed, and generation 2's identical reply was postedcompleted, no_send, so the user got nothing.
…lier send does not silence the reply - read_turn_records tested `turn_id in line` with turn_id None (an attempt with no turn), so one stale torn line raised TypeError on every delivery pass. - An earlier generation's room.message.send counts as already sent only when its outcome is succeeded, dispatched_unknown or not yet written; a failed send delivered nothing, so the new generation's reply is posted. - Smaller: the wake-unacked doc line names its flag; write_if_absent fsyncs the directory and names its temp `.tmp`; a busy journal lock costs one wait per close pass; a core reading moving but dead or abnormal counts as idle for the stall; docstrings state the torn-before-ids and deferred-close behaviour.
Collaborator
Author
|
Both items are fixed in c29b2dc (head c29b2dc). A (blocking): no crash with no turn.
B: a failed earlier send does not silence the reply.
C: docs. The protocol doc's wake line now names The smaller notes:
|
sonichi
approved these changes
Oct 30, 2026
Owner
There was a problem hiding this comment.
Approving at c29b2dc (agent-assisted, Blue for Chi's seat). Both items are fixed and re-probed:
(turn_id and turn_id in line)no longer raises for a no-turn attempt; another task's torn line is skipped, and one naming this task still makes the journal unreadable.- An earlier generation's send whose outcome failed no longer counts, so gen-2's reply posts
completed. A failed send followed by a successful one still counts as sent.
Minor: a failed outcome with no operation_id adds None to the failed set, so an id-less send intent would count as failed. That only matters if the proxy ever writes id-less records.
agent-turn (47) and sparrow-task-attempts (41) pass locally. CI was still running when I approved.
keweichen
requested changes
Oct 30, 2026
keweichen
left a comment
keweichen
left a comment
Collaborator
There was a problem hiding this comment.
Changes requested at exact head c29b2dc3372a9a71c40228fe78c6d2fabe2f301b — not ready to merge.
The earlier code findings are fixed and discriminated: journal close now requires the lock and retries; stall uses real core motion; failed result publication retries atomically; a no-turn torn line no longer raises; and failed/rejected sibling-generation sends no longer suppress the new reply. Focused suites pass. Keeping the new tests against 573004d0… reproduces the two sibling-send failures and both old no-turn TypeError paths.
Blocking: no exact-head post-restart beat/fence round trip exists. The body’s live runs name earlier builds, and the current evidence says this witness is still being scheduled. This changes the bridge/network/delivery loop, so harness proof cannot establish the shipped path. Worst case after opt-in is a valid task being fenced or failed incorrectly, losing or duplicating its answer.
Exact-head required CI is also red at review time. python standalone tests (8/9) reports failures in codex-core-launcher.test.py and watch-tasks-stream-bare-directory-event.test.py; the aggregate clean-install gate consequently failed. These look outside the changed layer, but the gate still needs a green rerun. CLA passes.
Unblock condition: paste the real restarted current-head beat/fence witness and restore required CI to green.
Collaborator
Author
|
On the red CI at c29b2dc:
|
Collaborator
Author
|
Live post-restart witness on the exact engine (run by Mark's coordinator agent on Mark's Mac, local k3d rig) Build: local DMG 0.7.99-agentactivity.17. Engine sutando Restart: the app was quit and relaunched on install (~04:45Z), the updater moved the engine to 64c0f17, core and both workers restarted; then: a. Normal reply, beat/turn round trip (mention 04:48:17Z: "reply exactly PONG-5883")
b. Cancel fence via beat (mention 04:48:33Z: "run
Not covered here: a broker-held |
bassilkhilo-ag2
approved these changes
Oct 30, 2026
bassilkhilo-ag2
left a comment
bassilkhilo-ag2
left a comment
Collaborator
There was a problem hiding this comment.
Approving at c29b2dc (agent-assisted, Bassil's Sutando).
keweichen's two remaining unblock conditions, checked
- Post-restart witness (comment 6515300534). I checked it against this head, not just its labels:
- c29b2dc is an ancestor of the witnessed engine 64c0f17 (
git merge-base --is-ancestor), as is #5941 2ed366a. - Every witnessed log line comes from this head's code: "Stop requested", "answered the Stop with nothing to deliver", "settled without a post (the Stop is confirmed)", "deliberate result file", "lease closed, not sent" all
git grepin c29b2dc. - Coverage: ack with lease and generation 1, a normal reply posted
completedwith its turn file closed under the lock, a beat hearingtask_cancelled7 s after Stop (consistent with the 15 s cadence the body said was not yet run live), and a cooperative Stop endingcancel_confirmedwith nothing posted. That is the beat/fence round trip asked for.
- c29b2dc is an ancestor of the witnessed engine 64c0f17 (
- Required CI is green at this head: 39 pass, 3 skipping (release/Windows only),
tsc + tests (clean install)and diff coverage pass.
Earlier code blockers: re-read at this head. close_turn (src/agent_turn.py:247-266) now changes nothing without the journal lock; write_if_absent (task_attempts.py:493-513) publishes via link of a fsynced temp, so a failed write leaves nothing behind and is retried; (turn_id and turn_id in line) and the failed sibling-send exclusion are in. Vendored copy is byte-identical.
Flag defaults: LIFECYCLE_V2 / TURN_CONTEXT (remote_gateway_bridge.py:457-462) are == "1" opt-in, _ATTEMPTS is None otherwise, and the wake guard (4812) is under LIFECYCLE_V2. The two always-on changes are now stated in the body.
Local: fresh clone at the head: agent-turn 41, sparrow-task-attempts 29, gateway-agent-activity-hooks 54, gateway-agent-activity-lifecycle 4, gateway-disposal-delegation 76, all OK. review-checks.sh --diff PASS.
Non-blocking notes
- The witnessed engine is not main + #5941 + #5883 alone: it is the test/agentactivity-dmg3 merge, which also carries the wake, stop-interrupt and answers-room-question branches, and those change
_resolve_stop,_deliver_result_payload,_post_ready_resultsand task_attempts.py (+244). For a non-wake task those deltas reduce to this head's behaviour (wake_itemis None, so_own_copy_existsfalls back to_delivered_copy_exists), so I accept the witness for this layer, but the comment's "plus this stack" undersells what was running. - Not witnessed live: a
stale_execution_generationfence (reclaim with a generation bump), thewaiting_inputrelease, the duplicate-sendno_sendbranch of the turn gate, anddispatch_lost. All are covered by the hook tests; worth one live pass before the flags are turned on for anyone else. - provider_ag2space.py:131-136: the always-on 409/400 settle returns
DeliveryOutcome.CONFIRMED. Fortask_already_terminalon a late reply (flags off, task timed out server-side) the ledger will read delivered while nothing reached the room; previously it was quarantined and visible. Theprovider_ref=settled:<code>keeps it auditable, but a distinct log line at the bridge would help whoever investigates a "missing reply".
Worst case for others by default: none beyond the two stated always-on settles; everything else is behind REMOTE_TASK_LIFECYCLE_V2=1.
keweichen
reviewed
Oct 30, 2026
keweichen
left a comment
keweichen
left a comment
Collaborator
There was a problem hiding this comment.
Re-reviewed exact REST head c29b2dc3372a9a71c40228fe78c6d2fabe2f301b. My existing changes-request remains; this head is not ready to merge. CI and CLA are now green, and the earlier code findings remain fixed and discriminated.
Blocking: the new post-restart witness is cumulative-stack evidence, not an exact-head or byte-identical executable witness. Comment 6515300534 ran engine 64c0f1746. Although c29b2dc3 is its ancestor, that engine is 164 commits ahead. Eleven of this PR's 17 paths differ, including src/agent_turn.py and its vendored copy, remote_gateway_bridge.py, and task_attempts.py. Their runtime delta is +511/-101 and changes the witnessed normal/cancel path, including _resolve_stop, _deliver_result_payload, TaskAttempts.finish, and result_file_decision; the build also contains later Stop-specific work.
The posted run is useful evidence for the cumulative AgentActivity stack, and its observed normal delivery plus beat/cancel sequence is internally coherent. But ancestor inclusion and matching log strings do not establish current-head behavior when the executable blobs on that path differ.
Unblock by restarting an artifact built from c29b2dc3 itself and repeating the normal reply plus cancel beat/fence round trip. CI does not need another rerun unless the head moves. Worst case otherwise is a later-stack change masking a current-head delivery defect, allowing a valid reply to be lost, duplicated, or incorrectly cancelled.
Contributor
|
@cla-assistant check |
Collaborator
Author
|
Exact-head post-restart witness at Method: the app's engine is a git checkout, so it was switched to this exact head, and then the gateway, core and workers were restarted, so every engine process runs this head's code. The app was not relaunched, because a relaunch would merge the release into a non-release head. The desktop proxy is main d30d82338's (turn binding, #936 gate, #1037). Lane a. Normal reply (05:07:38Z, "reply exactly PONG-5883") turns: b. Cancel beat/fence (05:07:56Z, "run turns: The only non-head code running is the compiled TypeScript under dist/ (voice, web client, conversation server), which is not on the gateway → core → result path. |
keweichen
requested changes
Oct 30, 2026
keweichen
left a comment
keweichen
left a comment
Collaborator
There was a problem hiding this comment.
Changes requested at exact REST head da80760 — not ready to merge.
The current-head live-path blocker is resolved. The posted checkout/restart chronology places the bridge and core starts after the tracked-clean da80760 checkout. The normal task completed, delivered once, archived, and closed its turn; the Stop was heard by the next beat, reached cancel_confirmed and runtime_confirmed, closed the turn, archived no-send, and posted nothing.
I checked the merge mapping rather than assuming tree identity. da80760 merges c29b2dc with current main. Sixteen of this PRs 17 paths retain the c29 blobs; only packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py differs, incorporating the merged shared named-lane ID validation. This witness exercised that integration through ag2space_dlocal, so it is current executable evidence rather than ancestor-only evidence.
Sole remaining blocker: required exact-head CI is red. python standalone tests (8/9) actually ran and failed the codex-core-launcher notifier timeout plus the fswatch EOF assertion; tsc + tests (clean install) consequently failed. These are the same two failure signatures from attempt 1 on c29b2dc, which passed on attempt 3, and neither failing path is in this PR diff. That supports treating them as unrelated flakes, but da80760 still needs its required green gate. CLA passes.
Unblock with a green unchanged-head rerun at da80760; then this changes request can be replaced with APPROVE.
keweichen
approved these changes
Oct 30, 2026
keweichen
left a comment
keweichen
left a comment
Collaborator
There was a problem hiding this comment.
APPROVED at exact current REST head da80760. The change is ready from the code and evidence standpoint. The PR remains draft, so mark it ready and recheck the current gate before merging.
The sole remaining block is cleared: CI run 41006258644 attempt 3 completed successfully at the unchanged head with 39 passes and 3 expected skips, including python standalone tests (8/9), tsc + tests (clean install), and diff coverage. CLA passes.
The exact-head post-restart witness remains applicable and covers normal DeliveryCore delivery plus the cooperative Stop beat through cancel_confirmed, runtime_confirmed, turn close, and no-send settlement. I also rechecked the base-only merge mapping: 16 of the 17 PR paths retain the reviewed c29 blobs; the remaining vendored bridge path picks up shared named-lane ID validation exercised by ag2space_dlocal.
The earlier journal-lock, real-motion stall, atomic publication, torn no-turn line, and sibling-generation findings remain fixed and discriminated. Worst case with the opt-in lifecycle enabled is a valid answer being lost, duplicated, or incorrectly cancelled; default-off containment, exact-head suites, and the live normal and Stop branches mitigate that risk. No review blocker remains.
marklysze
marked this pull request as ready for review
October 30, 2026 05:30
marklysze
added a commit
that referenced
this pull request
Nov 1, 2026
…[AgentActivity] (#5885) * feat(turn): immutable per-turn context file and 3-outcome turn gate Adds src/agent_turn.py (vendored into ag2-sparrow): the 0400 turn file at state/turns/<task_id>_g<gen>_<ulid>.json, its .closed fence, the journal reader for dispatch_intent/dispatch_outcome records, and classify_turn. A plain-text answer may be sent only when the turn has no bound intent, no unbound intent since it began, and core-api confirms zero operations. * feat(gateway): lease generation, per-task beats, cancel and the turn gate on results Behind REMOTE_TASK_LIFECYCLE_V2 (default off) the bridge sends one lease-holder worker_id and protocol_version 2 on ack, records the execution_generation the ack reports, beats each live attempt on POST /v1/tasks/heartbeat, and puts the generation and worker_id on every result. A fenced attempt (task_cancelled, stale_execution_generation) stops the core through a CANCEL_INSTRUCTION task and is cancel-acked; its late result is settled, never posted. Every five minutes GET /v1/tasks/active is reconciled with the in-flight set. With REMOTE_TASK_TURN_CONTEXT too, each attempt's turn file is written on ack and closed when its result settles, and the result is gated on the turn's journal: skipped when the task closed terminally through MCP, posted as waiting_input + no_send while suspended, and posted with no_send when the same text was already sent as a room message. A result 409 that ends the task, or a 400 waiting_input_without_blocker, settles the item instead of quarantining it. A re-served task carrying `wake` is left unacked rather than closed as a redelivery. * docs(gateway): the opt-in task lifecycle calls and turn gate * docs(changelog): the opt-in task lifecycle (#5883) * docs(src-map): list agent_turn.py * fix(gateway): register test_task_attempts in CI; name the ledger comparison apart from the delivery reconcile * test(gateway): cover the lifecycle hooks in process; unique control-task names The coverage gate measures tests/**/*.test.py in process: the package test moves to tests/sparrow-task-attempts.test.py, the bridge hooks get an in-process suite, and the vendored agent_turn.py copy is omitted like the other src/ copies. Two cancels in one millisecond no longer share a control-task name, and a recorded acked_at of 0 is read as itself. * feat(gateway): run a woken task as a new turn and ask in the room under turn context With REMOTE_TASK_TURN_CONTEXT, a task re-served with `wake` after its question was answered runs once per blocker_episode_id: it is queued again under the same id with each answer appended to the task's line as quoted room data, acked at the same generation into a new turn, and its result rides its own outbox item `<id>.wake-<episode>`. The earlier attempt's archived copies no longer settle it. Each task file also carries one writer-owned line telling the agent to ask a question that blocks the task with room.question.ask on its exact task_id. * test(gateway): a wake runs once in process, with its answer and its own result item * docs(changelog): resuming a woken task (#5885) * feat(gateway): name the chosen option's label in a woken task; log an unrun wake once per episode * test(turn): an outcome with null applied fields is unknown, neither terminal nor suspended * fix(gateway): a result waits for its lease, a fenced ack stops the core, a corrupt journal is unreadable Under REMOTE_TASK_LIFECYCLE_V2 a result written before its task's ack named the lease waits for that ack (at most 120 s), and records the proxy could not bind but that name the task count as the attempt's turn from publication on, so neither the lease fields nor the journal gate can be skipped by a fast core. Every fencing error_code on either ack path (404 or 409) fences the attempt, retires its pending ack and, for a published task, stops the core, except a stale older lease whose newer serve is pending here. A journal line that does not parse makes the journal unreadable instead of absent; an unterminated last line younger than 30 s defers the result. A served task carrying `wake` is left unacked whether or not this node saw its first attempt. A late ack for a task no longer queued opens no turn. * fix(gateway): an unanswered ack never releases an ungated result; a redelivery keeps the journal boundary On a broker that has ever named a lease, a result whose ack never answered keeps waiting until the ack lands, the attempt is fenced, or the server closes the task. On a broker never seen to name one, the 120 s fallback is gated on the journal records that name the task, so an early send, terminal action or blocker still rules. A task served again before its ack keeps its first publication time, so its earlier unbound room actions stay inside the turn's journal window. * fix(gateway): an id's archive and retirement are one step to a wake, on every settle branch The normal, skip-marker, dedup and orphan-recovery branches archive a result and then retire its id (in-flight removal, room and media sidecars). Each now runs that whole transition under the wake handoff lock and marks the id retiring, so a wake served in that window, on another thread or this one, waits to be served again instead of being published and then losing its tracking. * docs(gateway): one locked step for archive and retirement * fix(gateway): an attempt whose ack never answered ages into the ledger reconcile from its first publication A result held because its ack never named the lease on a broker that names leases is now released by the reconcile too: once the complete active listing omits the task past the grace, the attempt is fenced, the core stopped, and the result settled without a post. * fix(gateway): the disposal judgement is inside the retirement step, and a settle never archives an open wake The no-live-result recovery branch now judges disposal inside the same locked retirement step as its settle, so a wake admitted after the judgement waits to be served again. A settle also re-checks inside the lock: while a wake for the id is open here and its own result is not delivered, nothing is archived or retired, whatever the caller judged before it took the lock. * fix(gateway): a Stop is cooperative and never drops a finished reply; every held task beats A task_cancelled in a per-task beat no longer fences the attempt. The core gets one CANCEL_INSTRUCTION task; once it has handled it, a result it wrote for the task is posted normally with no cancel-ack, and with no result the stop is confirmed on cancel-ack and a later result is settled. A task_cancelled on the ack still means the task is not started and is cancel-acked at once. The per-task beat now runs on its own 60 s cadence ahead of the client heartbeat's backoff, so a Stop reaches every task the bridge holds for as long as it holds it. * fix(gateway): a Stop answered with nothing to deliver is confirmed as stopped, not no_action_needed While a Stop is pending, a core result that is no_send or empty confirms the Stop with a cancel-ack and is settled locally; it is never posted as no_action_needed. A failed cancel-ack holds the result for the next pass. A result with a body is still posted (a finished reply is never dropped). * fix(gateway): a Stop settled by its result passes the attempt snapshot, like every other settle * feat(gateway): a held task beats every 15 s so a Stop is heard within ~15 s TASK_HEARTBEAT_INTERVAL_S (15) replaces the 60 s client cadence for per-task beats; the client heartbeat stays at 60 s. While an attempt with a generation is held, the task long poll returns when the next beat is due, so the beat is not delayed by a 25 s poll. Attempts without a generation still never beat. * fix(gateway): an attempt whose task left the queue unanswered is failed after 10 min, not beaten forever On every per-task beat, an attempt with a generation whose task file is no longer in the queue (archived or removed), with no result file, no claim and no turn activity from the session's own hooks for DISPATCH_LOST_S (600 s), is marked dispatch_lost: its beats stop and a [no-send] result file is created only if none exists (O_EXCL, so a real answer always wins). The result posts with status failed and no_send. A file put back in the queue resets the clock. * fix(gateway): a queued task the core took up and then left is failed after 30 min A turn that ends without a result leaves its task file queued, so the per-task beat renewed the lease forever and the room showed the task as working. Once a turn row newer than the file exists and nothing more is written for 30 min, the attempt stops beating and posts failed + no_send. A file the core has not reached yet keeps beating. * fix(gateway): a wake turn ends with its own result A wake reuses the task's filename after the first turn's result was archived. The core read it as a re-emitted, already-answered task, recorded a turn-ledger no-send and ended the turn with no result, so the task stayed working. The wake's file now says it is a new turn; the Stop hook no longer takes an archived result older than the queued file as its answer; and a turn that recorded a no-send loses the in-progress exemption. * fix(gateway): an attempt the broker still holds is failed, not dropped from flight The in-flight reconcile dropped an id whose task file was archived without a result as "completed elsewhere", even while its attempt still beat. Its lease then lapsed, the broker re-served it, and the cycle repeated with nothing posted. A beating attempt is now left to the 10-min dispatch watch, which posts it as failed with no_send. * fix(gateway): an older generation's cancel never stops a newer one served here A stale_execution_generation fence queues a CANCEL_INSTRUCTION naming the local task id. When the broker then re-serves a newer generation of that task to this node, the old cancel would stop the new attempt. Acking the newer generation now withdraws that cancel if the core has not taken it yet, and logs when it already has. * fix(gateway): a newer attempt stopped by an older generation's cancel is posted as failed When the core took the older generation's cancel before the newer generation was acked, the cancel could not be withdrawn. Once the core has handled it, a newer attempt left with no result and no queued task file is now posted as failed with no_send at once, instead of waiting out the 10-min dispatch watch. One the core still answers posts its own result. * fix(gateway): the cancel withdrawal passes the disposal and log-sink guards It removes only a control task the bridge wrote, so it joins the known movers; its OSError is logged through diagnostics.describe, not raw. * fix(ag2-sparrow): a task the broker closed leaves the inbox untaken, or stops the core A task_already_terminal or task_not_found fence from a beat, an ack or the active-list reconcile withdraws the task file when the core has not taken it up (no turn row newer than the file), and otherwise queues one cancel instruction. The broker's reason is logged. Its result stays settled, never posted. Co-Authored-By: Claude <noreply@anthropic.com> * fix(ag2-sparrow): the ask-in-room rule sends visible text with room.message.send first A turn that leaves its question open posts its result as waiting_input + no_send, so a reply written to the result file never reaches the room. The task file now says so, and says to send anything the room should read with room.message.send before asking. Co-Authored-By: Claude <noreply@anthropic.com> * fix(agent-activity): a task cancelled on its ack is withdrawn if the core has not taken it A task_cancelled answer on the ack now goes through _withdraw_closed, like a closed task: an untaken task file (a woken one included) leaves the inbox so the core never starts it, and only a task the core has taken up gets a CANCEL_INSTRUCTION. Its result stays settled. * fix(agent-activity): a woken attempt is answered only by its own result The first attempt's archived reply shares the task id, so withdraw-on-close, dispatch-lost, stall and Stop checks read a wake as answered. One helper, _own_copy_exists, judges a woken attempt by its own outbox item: a wake stopped before the core takes it is withdrawn, and a lost wake is failed. * fix(gateway): the Stop hook's wake handling applies only to a wake turn A wake's task file now carries a wake_episode header above task:. The Stop hook ignores a result older than the queued file, and drops the in-progress exemption after a turn-ledger no-send, only for such a file; every other task is judged exactly as on main. * fix(agent-turn): a turn's .closed fence is written under the proxy's journal lock close_turn takes <state>/room-actions.jsonl.lock with the proxy's format (O_EXCL, "<pid> <token> <start_ms>"), breaks it only for a dead pid and waits at most 2 s for a live holder, so a close never lands between the proxy's closed-check and its intent append. Past the wait the turn is still fenced, and the bridge logs it. Only turn-context attempts have turns to close. * fix(gateway): address the #5883 review: no fence on absence, torn lines, generations, stalls, locked closes - Reconcile: not being listed as outstanding no longer fences an attempt; only a beat's or ack's explicit terminal answer does. The active list is read as one page of 200; a longer listing skips the pass instead of walking a cursor. - A torn journal line fails only the turn or task it names; another task's torn line is skipped. - A reply sent as a room message under an earlier generation of the same task counts as already sent. - Stall: never while the core's health row reads moving (mid-turn); 30 min when it reads idle; 2 h when its state is unknown. - close_turn writes the .closed marker only while holding the proxy's journal lock; otherwise nothing changes and close_finished_turns retries it on the beat cadence, from the still-open turn file. - A failure result is written to a temp file and linked into place, so a failed write leaves nothing behind and is retried. - A task carrying `wake` is left unacked only with REMOTE_TASK_LIFECYCLE_V2=1; with the flag off it is handled as on main. * fix(gateway): the failure-result writer lives with the attempt lifecycle The bridge may not hard-link (its links are reserved for result disposal), so create-if-absent moves to task_attempts.write_if_absent; the bridge keeps only its logging. Also trims the stall comment to two lines. * fix(gateway): an owner-only question goes to the DM; a lost ask is never posted as the reply The ask-in-room rule applies the same audience test as a reply: a question about the owner's own errand, or data from their accounts or device, goes to their DM with ask-owner.py even from a shared room. A room.question.ask whose outcome is missing or dispatched_unknown now posts waiting_input with no_send, so the 'I asked X' note is never the completed reply; a broker with no open question refuses it, and the lease lapses so the task is served again. * fix(agent-turn): no crash on a torn line without a turn; a failed earlier send does not silence the reply - read_turn_records tested `turn_id in line` with turn_id None (an attempt with no turn), so one stale torn line raised TypeError on every delivery pass. - An earlier generation's room.message.send counts as already sent only when its outcome is succeeded, dispatched_unknown or not yet written; a failed send delivered nothing, so the new generation's reply is posted. - Smaller: the wake-unacked doc line names its flag; write_if_absent fsyncs the directory and names its temp `.tmp`; a busy journal lock costs one wait per close pass; a core reading moving but dead or abnormal counts as idle for the stall; docstrings state the torn-before-ids and deferred-close behaviour. * fix(gateway): the DM route says what this task's result is; a reconciled failed ask does not park The ask rule's DM route now ends the task with a [no-send] result and a one-line note. An ask whose reconciled outcome is known (failed) no longer parks the task as waiting_input; only an ask whose outcome is still unknown does. * fix(agent-turn): a failed ask posts failed, never its 'I asked' note An ask whose outcome is a failure, in the journal or from reconciliation, means no question exists, so the result posts failed with no_send instead of the note as the completed reply. An ask whose outcome is still unknown keeps posting waiting_input. * fix(gateway): a settled wake keeps its outbox item until its task leaves flight finish() forgot the wake item as soon as the wake's result was confirmed, before the caller archived the still-live result. A restart (or a failed archive) then re-offered that result through the first attempt's delivered item, which quarantined it to undelivered. The item is now kept in relay-wake-items until the startup sweep sees the task out of flight, and a re-offer through a settled wake's own item is recognised as delivered rather than re-sent. * fix(gateway): a wake attempt is retired only after its identity is saved finish() ignored each write. It now saves the wake's episode as seen, then its outbox item, and only then removes the attempt; any failed write returns False with the attempt kept, so the identity is never lost to a half-done finish. * fix(gateway): a wake that cannot be saved is not run; a failed retirement keeps the result live - open_wake() saves the opened wake itself and returns None when that write fails, so _write_task leaves the wake unacked and unqueued for a re-serve. - Every settled branch of _deliver_result_payload goes through _retired(): when finish() could not save the wake's identity it returns False, so the result stays live and in flight and the next drain retries. finish() now returns False only for a failed write; an attempt that moved on is True. - A wake's outbox item is its episode's own, so a re-offer after a delivered one is recognised as delivered and never re-sent. * fix(agent-activity): an unreadable attempt record never retires a wake The ledger, wake-episode and wake-item reads are tri-state: a missing file is empty, an unreadable or malformed one is None. finish() returns False on None, so the result stays live and in flight instead of being read as "the attempt moved" and archived without the seen episode or wake item. open_wake() leaves the wake unacked on None, so an unread episode list never re-runs a wake. * fix(agent-activity): an unreadable attempt record is never rewritten or delivered past _update() and sweep() leave an unreadable ledger or wake-item file as it is instead of writing over it from an empty read, so another task's write cannot erase the attempt a failed retirement kept. wake_item() returns UNKNOWN when the records that pick the result's item cannot be read; the result is then held (no POST, no item chosen), counted as undisposed and as an open wake. A path that cannot hold a file (its parent is not a directory) still reads as empty. * fix(agent-activity): a task is queued and acked only once its attempt is recorded remember_task() reports whether the attempt was saved; _write_task leaves a task it could not record unqueued and unacked for a re-serve. The pending-ack record is retired only after record_ack's generation is on disk: an accepted ack whose generation was not saved keeps its pending ack, and the pending-ack retry re-acks and records it, so the result posts under its generation and lease holder instead of as legacy. * fix(agent-activity): every attempt-ledger writer reports a failed write and its caller retries ack_first: _post_ack_first answers "retry" when _record_attempt could not record the ack, so the task stays staged and its manifest entry live. _record_attempt checks on disk that the ack wait ended, the generation and the broker's lease contract were saved, and an older generation's cancel was recorded as taken before it is withdrawn. Writers now return whether they saved: ack_settled, mark_dispatch_lost, fence (unsaved verdict), request_stop, stop_outcome, cancel_ack and the legacy fallback in result_hold. Callers act only on a saved write: a fence or Stop records its control task's id in the same write and queues it after; an unrecorded fence keeps its pending ack; a not-leased ack keeps it until its wait is ended; a dispatch-lost failure is written only once recorded. A worker id that cannot be saved is kept for the process. * fix(agent-activity): work a saved decision owes is retried until done, never dropped A fence, a Stop, a closed task and a dispatch-lost failure each save what they owe in the decision's own write (owed_ctl with the control's id, owed_close, or the failure body in dispatch_lost). _settle_owed carries it out and clears the marker only once the work is done; it runs right away and on every beat, so a control or result whose write failed is written when storage recovers, exactly once. The rollbacks that could erase the only retry are gone. record_ack re-saves a missing broker contract before its same-generation return, so a re-ack repairs one failed contract write and an ack_first task is published. A taken stale cancel keeps its marker until the failure it owes is recorded. Log lines no longer carry the ack's generation. * fix(agent-activity): owed work belongs to a generation and is settled from the ledger Round-10 review: a design pass over the work a saved decision owes. - Every debt (owed_ctl, owed_close, dispatch_lost) is stamped with the record's generation in the write that creates it. A newer generation is recorded only after the older one's queued cancels are withdrawn (renamed aside, so a crash cannot be misread as the core taking them) and its unposted failure result is archived through the disposal owner; the same write clears the older debts. - The retry owner is the ledger: each beat and each start upgrades, scans owing records, holds them in flight and settles them; _broker_holds counts them, so the abandoned reconcile cannot drop one. - An unreadable in-flight set at start retires nothing; the sweep never drops a record that owes. - A dispatch-lost marker rules only on the failure result it wrote, matched on the result file's own text; an answer the core wrote first posts as its answer. - TaskAttempts.upgrade() normalises records written by the base bridge once (schema 2): boolean dispatch_lost, a Stop or stale fence whose control was never queued, a closed fence with no marker. - Regressions for each case go through production paths and fail on 647f7b6; the Stop double-fault test fails on ac2b11f. * fix(agent-activity): one writer per attempt transition; unknown is held; migration at start Every read -> effect outside the ledger -> write on a task attempt runs under TaskAttempts.lock: ack recording (retract + record_ack), wake opening, settlement, beat fences and Stops, the per-task beat checks, the result ruling and finish. Settlement acts only on debts of the generation the record holds, never while a newer serve awaits its ack; any move to another attempt (including the first generation after a pre-ack fence, and a wake) retracts the earlier one first. finish compares execution_generation and attempt too. A failure result is owned through the file the bridge created: its identity is recorded before link() publishes it, replacing body equality. Unknown never becomes deletion: owing()/records() are tri-state, holds() keeps every id while the ledger is unreadable, adopted debts are settled only once the in-flight write committed, and an unreadable in-flight set is rebuilt from every attempt record. Legacy records move to task_attempt_migrations.migrate(), run once at start before settlement and the sweep; it adopts or retires cancels the base queued without recording, and a migration that did not commit sweeps nothing. Start brings attempts with a queued task or live result back in flight first. * test(agent-activity): cover the migration's unreadable, withdraw-failure and older-marker paths, and the start's unknown holds Through task_attempt_migrations.migrate and the bridge's start, ack, wake and retraction paths: no production source changes. * fix(agent-activity): the file ruled is the file published; serves and late acks join the transition - Result delivery holds the attempts lock through the local publish and, with lifecycle v2, publishes nothing of a result file that was withdrawn or replaced after the drain read it; the next pass rules its successor. - remember_task takes the attempts lock. A late ack for an older generation, or for an attempt already retired, records nothing; an ack refusal about a lease older than the recorded generation fences nothing; a beat fence is left to the pending ack while a newer serve awaits it. - An unreadable ledger during the result ruling holds the result; a ledger entry that is not a record is held and never swept. - taken_ctl watches every older cancel the core took, not only the last. - Migration finds controls under every id the bridge mints (a named instance's underscore included) and in .assigned-/.claimed- names; a control that moves while scanned restarts the scan; a missing queue under a missing workspace, or an unreadable live result, is unknown; cleaning up a withdrawn cancel never fails the start. - Startup discovers queued/claimed tasks and live results tri-state before migrating, settling or sweeping: a read fault is not absence. * fix(gateway): claim the result before any effect, bind serve and lease, one typed attempt reader The drain pins the result generation before its first effect and every effect reads the pinned bytes. remember_task records the lease with the serve, and acks, settles and fences act only for that serve. TaskAttempts.entry() returns record, absent or UNKNOWN for every reader. The migration adopts existing controls and never mints a new Stop. A retraction withdraws an unsent failure from the outbox. * refactor(gateway): drop the attempt-record migration; the ledger moves to relay-task-attempts-v2 Lifecycle v2 is opt-in, so records in the earlier shape exist only on opted-in rigs. The bridge no longer reads or upgrades them: the ledger has a new file name, and start logs that the earlier one is ignored. Removes task_attempt_migrations, _legacy_failure, SCHEMA and the holds that waited on a migration. * fix(gateway): a staged ack_first ack is recorded under the lease it was served with Without the lease, same_serve refused the ack of a leased serve, the attempt kept awaiting_ack and its reply was held. * feat(gateway): one transition owner for task attempts; claimed effects; runs bound to one generation Every change to a task's attempt goes through transition(task): a per-task lock, and a leaf commit under the ledger's file lock that re-reads, compares, writes durably and fails closed. A remote effect is claimed in one transition, sent with no lock held under its operation id, and committed only while the exact claim owns the exact run and generation. Each start takes the ledger over with a new epoch; an older bridge makes no further claim or call. A served task runs as a local run (<local id>~~<n> after the first), allocated before its file is published. A run's generation is fixed once it can execute: a serve under a new lease is held until its ack names the generation; the run's own generation moves the lease onto it, a newer one stops the run, reconciles its claimed effects and publishes a run of its own. A result with no live run is quarantined. Results are delivered on a worker per task. * fix(gateway): log no response value, name the sidecar mover, keep the witness launcher out of tests * fix(witness): the fault launcher reports its rule as JSON and matches a claim's lease where it lives The identity line held PosixPath values and json.dumps raised at start. A claim no longer carries the lease: an ack's claim names its serve, and any other effect's lease is the record's. The matched claim's whole identity is reported. Tests now drive the real bridge and main(). * fix(gateway): a newer generation's run answers on its own outbox item; run ids never reused A run a newer generation gets publishes its result on its own item (<task>.run-<n>), as a wake's does, so an earlier run's delivered item never swallows it. Before the switch, an earlier run's result claimed but not yet in the outbox counts as possibly sending: the switch waits, and a ruling held for the pending serve lets its claim go. A run id with anything on disk is skipped, and run counters are trimmed only for tasks the ledger no longer holds. Also: ledger ownership is retried each pass; result workers edit the in-flight set under its lock; a task's lock is kept only while a thread uses it; the wake lifecycle test follows the wake's own run id. * docs(gateway): a newer generation's run has its own outbox item; the switch waits on an unsent claim * test(gateway): every failure path of a held serve's ack, the switch and its reconcile; the launcher's crash and load; src run ids * fix(gateway): a result claim whose file the drain disposed of unpublished is ended in one place, so a newer generation never waits on it * fix(gateway): the ledger keeps insertion order, so trimming forgets the oldest; a just-settled or live task's wakes are kept * fix(gateway): a result claim is ended only once its exact reply is gone unpublished; each disposal path carries on to generation 2's answer * fix(gateway): a name or outbox item that cannot be read is never taken as absent; a kept result claim logs what clears it * fix(gateway): a result claim records each outbox item before it is published, so a switch settles exactly those; a run's item has one name for every reader * docs(catalog): remote-gateway-protocol.md verified 2026-11-01 * fix(gateway): a generation served after the earlier one finished is held for its ack, never taken for a redelivery; every run after the first has its own item * test(witness): the reclaim fault takes the bridge offline until the broker re-leases, and posts the task's result only once the newer serve exists * fix(gateway): an accepted ack counts as done only once its generation can reach the core; a newer lease is always held; a held serve is saved under its broker id * fix(gateway): a newer generation's task says whether the earlier answer was delivered, so the core never takes it for a redelivery it answered * fix(gateway): a newer run is published only once what became of the earlier answer can be read, decided in one read * fix(gateway): the owed run and the earlier run's delivery state are read from one ledger snapshot --------- Co-authored-by: Claude <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed, and why
Sutando's side of the AgentActivity Rev 2.6 task lifecycle, behind two opt-in settings that are off by default. With both off, the wire and behaviour are unchanged except for the two default-on safety fixes listed under "Always on".
REMOTE_TASK_LIFECYCLE_V2=1(sparrow bridge)worker_id(stable per install) plusprotocol_version: 2, and records theexecution_generationthe 200 reports.POST /v1/tasks/heartbeatevery 15 s (TASK_HEARTBEAT_INTERVAL_S); the client heartbeat stays at 60 s.execution_generation+worker_id.task_cancelled/stale_execution_generation, from the ack or a beat):CANCEL_INSTRUCTIONtask (newsource: relay-control; its result is retired locally, never posted);GET /v1/tasks/activeis reconciled with the in-flight set every 5 min.REMOTE_TASK_TURN_CONTEXT=1(also needs the first)state/turns/<task_id>_g<gen>_<ulid>.json(0400) is written on ack. The desktop MCP proxy binds room actions to it..closed) when the result settles. Orphans are closed at start.state/room-actions.jsonl):applied_terminal)waiting_input/input_ready/ open blocker)status: waiting_input,no_send— nevercompletedroom.message.sendin the turn carried the same text (body_sha256)status: completed,no_sendstatus: failed,no_sendstatus: completedwith the bodyAlways on, intended (safe on old brokers, which never send these codes or fields)
409 task_already_terminal | stale_execution_generation | task_cancelledor400 waiting_input_without_blockersettles the outbox item instead of quarantining it as undelivered.source: relay-controltasks (the bridge's own cancel signals, which only exist under the flag) instead of posting them.Behind
REMOTE_TASK_LIFECYCLE_V2=1too: a served task carryingwakeis left unacked (never closed as a[no-send]redelivery, which would lose the human's answer; running it is the stacked follow-up PR). With the flag off a wake is an ordinary task, as on main.Review first
src/agent_turn.py— turn file, journal reader,classify_turn,result_file_decision. Vendored byte-identical aspackages/ag2-sparrow/ag2_sparrow/agent_turn.pyviatools/sync_from_src.py.packages/ag2-sparrow/ag2_sparrow/task_attempts.py— attempt store, beats, fence/cancel-ack, reconcile. Package-canonical; only the bridge uses it.packages/ag2-sparrow/ag2_sparrow/remote_gateway_bridge.py— thin hooks: ack (both paths),_post_heartbeat,_deliver_result_payload(decision + fields + close on both success branches), orphan sweep (relay-control), wake guard, startup sweep, ledger reconcile.packages/ag2-sparrow/ag2_sparrow/delivery_core/provider_ag2space.py— the settled answers.Deviations from Rev 2.6
Logged with the coordinator in
ag2space/dev_docs/agent_activity/deviations.md.A2 / T12 — needs Mark's sign-off. README invariant 6 says Case 1 stdout auto-send happens only when zero
dispatch_intentrecords exist. Sutando has no stdout. Its result file is a deliberate agent output, so it is posted unless the task is already terminal, or the turn holds aroom.message.sendintent with the samebody_sha256. That check counts succeeded, unknown and unmatched outcomes alike, so it fails closed on possible duplicates.sutando/relay_worker.py,tui_hitl_bridge.py,task_reconciler.py,prompts/src/agent_turn.py(vendored),ag2_sparrow/task_attempts.py, bridge hooksAG2_TURN_CONTEXT_FILEin the child envtask_idglob (T3)failed+no_sendwaiting_input+no_send, nevercompleted(architect guardrail a)task_cancelledis a cooperative Stop. The core gets aCANCEL_INSTRUCTIONtask and nothing is fenced: a reply it still writes is posted with no cancel-ack, and without one (or with ano_sendor empty result) the stop is cancel-acked. An ack'stask_cancelledmeans the task is not started, and it is cancel-acked. A stale generation still fences the attempt and stops the corephase/step_summarytask_id,worker_id,execution_generationonlyroom.question.ask(source="runtime_tui")SKILL.mdwould post "I asked X" as the final reply with the flags offtask_status=donetask_statusTASKS.mdGET /v1/tasks/active; extra server tasks listed instate/relay-outstanding-tasks.jsonTASKS.mdexists; no prompt hook for an active-task context yetUser behaviour this proves
With the flags on against a v2 broker:
waiting_inputinstead of being closed.Docs
docs/remote-gateway-protocol.md: two config rows and a "Task lifecycle (opt-in)" section.docs/catalog.jsonlast_verifiedis bumped.Before / after
Before —
72c3fb29c(origin/main when this branched; since rebased ontoe369bf916, whose 7 new commits touch none of these tests' paths), with only this PR's tests copied in:After — HEAD
2ac98ba8b(the new tests plus the existing turn-ledger and sparrow vendoring tests; re-run after the rebase):Tests run
tests/agent-turn.test.py(26),tests/sparrow-task-attempts.test.py(20; first written aspackages/ag2-sparrow/tests/test_task_attempts.pyand moved so the coverage gate measures it),tests/gateway-agent-activity-lifecycle.test.py(4, a real bridge process against the mock relay on a socket),tests/gateway-agent-activity-hooks.test.py(11, the bridge hooks in process). The spec's required cases:test_immutable_turn_context_file,test_turn_gate_suppresses_stdout_when_dispatch_intent_exists,test_turn_gate_durably_suspended_on_waiting_input_and_202,test_heartbeat_409_aborts_turn,test_tasks_md_server_reconciliation. Architect guardrails:test_a_suspended_turn_is_never_posted_as_completed,test_a_turn_closed_terminally_through_mcp_skips_the_result_file.tests/gateway-*,tests/sparrow-*,tests/*bridge*,tests/turn-ledger*(187 files, run as scripts): 185 pass. The 2 failures,tests/discord-bridge-access-mutation-call-sites.test.pyandtests/discord-bridge-send-body-file.test.py, fail identically on the parent72c3fb29c, so they pre-exist and are unrelated.packages/ag2-sparrow/tests/test_*.py, per file: 32 pass.test_cloud_session_interlock.pyfails 4 here and 21 on the parent;test_task_lifecycle.pyfails the same 2 on the parent (test_a_flat_load_finds_its_siblings_through_the_path,test_unmodified_main_refuses_the_fenced_work_dir_at_startup_with_everything_unchanged); both pre-exist.test_wheel_contents.pycollects nothing under pytest; run as a script it passes, including "installed wheel imports".ci-covers-every-python-testfailed: the package test wasn't run by CI. Fixed by the move above.delivery-core-enforcementfailed: a method namedreconcilemust have the delivery contract's signature. Renamed tocompare_with_server.diff coverage >= 95%failed: the hooks only ran in bridge subprocesses, which aren't measured. Fixed with the in-process suite, and the vendoredagent_turn.pyis now omitted like the othersrc/copies. Measured locally with the gate's.coveragerc+diff-cover: 98%.tests/skills/worker-pool/pool-restart.test.pyfails on origin/main too.tests/skills/process-hygiene/reap-orphans.test.pypassed locally on this branch and on main, so it looks flaky.acked_atof 0 was read as missing.python3 tools/sync_from_src.py --check: in sync.ruff checkon the changed files: clean.git diff origin/main | bash scripts/review-checks.sh: PASS.Live round trip
Run on the local k3d rig with the flags on; see Live evidence: local k3d rig below. The cancel path also ran on desktop test build .14 (engine
81d0c7397, 2026-10-30): Stop on a running task returned 200cancel_requested, the bridge loggedStop requested, the ledger reacheddead/cancelled_by_user/cancel_confirmedabout 30 s later, and nothing was posted to the room. Both flags are off by default, so merging changes nothing until they are set; turning them on againstdev.ag2.spaceneeds backend B1/B2/B4 deployed there.Merging does not publish the package;
publish-sparrow.ymlis a separate step.Turn files and the proxy's journal lock (
2d8dd86f5)mkstempinstate/turns/, write,fchmod 0400,fsync, thenlink()into place (which refuses an existing name), then the temp file is unlinked and the directory fsynced (src/agent_turn.pywrite_turn_context).close_turnwrites the.closedmarker while holding the proxy'sstate/room-actions.jsonl.lock: sameO_EXCLfile and"<pid> <token> <start_ms>"format; a dead pid's lock is broken, a live or half-written one is waited on for at most 2 s. So a close can't land between the proxy's closed re-check and its intent append. Past the wait the turn is still fenced, and the bridge logsclosed without the journal lock. Only turn-context attempts (REMOTE_TASK_TURN_CONTEXT=1) have turns to close.tests/agent-turn.test.py,CloseUnderJournalLock): the marker is written while this process holds the lock and the lock is released; a live holder delays the marker until it releases; past the wait the holder's lock is untouched and the turn is fenced; a dead holder is broken; a half-written lock is waited on; the default wait is 2 s.src/agent_turn.pyis at 100% line coverage.Stacked child: #5885 (wake / resume). Merge this first.
Review response (adversarial review of
3bd808b63)Each finding was reproduced with the reviewer's script, fixed in
db5e0938a, and pinned by a regression test. The reviewer's script, run against the fix:test_a_result_written_before_the_ack_waits_for_the_lease,test_room_actions_before_the_ack_still_gate_the_result,test_an_ack_that_never_names_the_lease_holds_the_result_only_so_long,test_a_late_ack_for_a_finished_task_opens_no_turn,test_unbound_records_naming_the_task_belong_to_its_turnerror_codeon either ack path (404 or 409: stale, cancelled, not found, already terminal) fences the attempt, retires the pending ack and, for a published task, stops the core. The exception: a stale older lease while a newer serve of the same task is pending here, which that serve's ack decidestest_every_fencing_ack_answer_fences_retires_and_stops_the_core,test_a_stale_older_lease_with_a_newer_serve_here_is_left_to_the_newer_ack,test_a_fenced_ack_first_task_is_refused_without_reaching_the_core,test_a_legacy_not_leased_refusal_releases_the_result_without_fencingfailed+no_send). An unterminated last line younger than 30 s is an append in progress: the result is deferred and retried; once older, it is corruptiontest_a_corrupt_journal_line_is_unreadable_not_absent,test_an_append_in_progress_defers_and_a_stale_one_is_corruption,test_a_journal_line_still_being_written_defers_the_resultwakeis left unacked, decided before any local shortcuttest_a_wake_is_never_run_without_support_even_as_the_first_sight_of_its_idThis branch also merges origin/main (the only conflict was the ack timeout constant; both changes kept), which clears the
CONFLICTINGstate.Live evidence: local k3d rig (post-restart round trip)
Run on a Mac with the desktop test build and engine
15f5df6b4(an earlier #5885 commit, which includes this PR's lifecycle code).REMOTE_TASK_LIFECYCLE_V2=1andREMOTE_TASK_TURN_CONTEXT=1were set for the local lane. The rig is a local k3d cluster running backend PR #2953 and MCP #112, not dev.ag2.space.relay-task-attempts.jsonandrelay-lease-worker-idwere created. The task was served with a lease and delivered, and the PONG came back in 9.2 s..closedfence on exit. The bridge loggeddeliberate result file -> {'status': 'completed'}, and the PONG came back in 9.1 s.room.question.ask, which shows the per-task ask-in-room rule reached it. Owner health showedwaiting_input. After the answer it showedinput_ready, thenworkingwithin 2 s, with the wake re-served and acked and a new turn opened. The reply "You chose Blue" was responded and the task cleared.The dev.ag2.space run follows once backend #2953 is on dev.
Rerun on
21d87b0c0(the #5885 head before the Stop fix below; new DMG, engine fast-forwarded,REMOTE_TASK_LIFECYCLE_V2=1+REMOTE_TASK_TURN_CONTEXT=1, same local k3d rig with backend #2953):waiting_input, answer,input_ready, thenworking2 s later. "You chose Blue" arrived at 18.4 s.task-0e542ed29e181e5510; the core was told tosleep 150and then reply). The owner pressed Stop at 03:25:58, which gavecancel_requestedat generation 1 (no bump). The per-task beat saw it at 03:27:01 (about 63 s) and loggedStop requested — the core is asked to stop…. At 03:28:12 the core wrote a[no-send]result. The bridge posted{status: no_action_needed}, the ledger settled it SUPPRESSED, and no cancel-ack was sent. That is a bug: a Stop the core obeyed was recorded as "no action needed" instead of cancelled.Fix (
dbd53b6b1on #5883, merged into #5885 ase436b5b90): while a Stop is pending, a result that isno_sendor empty confirms the Stop. The bridge sends the cancel-ack and settles the result locally; it is never posted. A failed cancel-ack holds the result for the next pass. A result with a body is still posted, so a finished reply is never dropped. Before (parentbdb53cbee) and after,python3 tests/gateway-agent-activity-hooks.test.py:Stop rerun on
e436b5b90: PASSED. The flow wascancel_requestedat generation 1, then the beat saw it, then the bridge loggedthe core answered the Stop with nothing to deliver — stopped, thencancel_confirmed/ dead /cancelled_by_user. This verifies the no-send fix live.Per-task beat now every 15 s (
5bf7fd3acon #5883, merged into #5885 as9b241f374). The ~63 s it took the Stop above to be heard was the 60 s beat cadence.TASK_HEARTBEAT_INTERVAL_S = 15now pacesPOST /v1/tasks/heartbeatfor held attempts with a generation, while the client heartbeat stays at 60 s. While such an attempt is held, the task long poll returns when the next beat is due, so a 25 s poll cannot delay it. Inpython3 tests/gateway-agent-activity-hooks.test.py, before is the parentdbd53b6b1and after is the fix:The 15 s cadence has not been run live yet.
An attempt the core will never answer is failed, not beaten forever (
52090712c)Live finding: after an app restart, the core's startup orphan check archived two freshly served tasks unanswered (fixed separately in the task-orphan-check PR). The bridge still held them, so the per-task beats kept them
workingindefinitely. Now, on each beat, an attempt with a generation that meets all of these is markeddispatch_lost:DISPATCH_LOST_S).Its beats then stop. A
[no-send]result file is created only if none exists (O_EXCL, so a real answer written in the same instant wins). The result posts withstatus: failedandno_send, which ends it visibly (dead). A task file put back in the queue resets the clock.python3 tests/…at the parent12dbd4ce7and at HEAD:CI
All checks are green, including
tsc + tests (clean install)(run 40257037250). Earlier red shards were pre-existing onmain:gateway-stall-exit-inflight-sendwas fixed by #5887 and is merged in;pool-restartpassed on this run (flaky on main run 40233569375).