Repository navigation
feat(worker-pool): the tested multi-worker behavior, consolidated into the skill - #4209
Conversation
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 90%. Diff CoverageDiff: origin/feat/startup-worker-section...HEAD, staged and unstaged changes
Summary
skills/worker-pool/scripts/pool_delivery.pyLines 282-296 282
283 if a.command == "payload":
284 # A sentinel names its payload and holds none; the mapping lives here
285 # so a caller never re-spells tasks/.
! 286 if not a.sentinel:
! 287 ap.error("--sentinel is required for payload")
! 288 parsed = parse_sentinel(a.sentinel)
! 289 if parsed is None:
! 290 ap.error(f"not a delivery sentinel: {a.sentinel}")
! 291 print(payload_path(ws, parsed[0]))
! 292 return 0
293 if a.command == "residue":
294 if not a.task_id:
295 ap.error("--task-id is required for residue")
296 print(residue(ws, a.recipient, a.task_id))skills/worker-pool/scripts/pool_route_handler.pyLines 116-131 116 return MUST_HANDLE
117 if args.probe:
118 return code
119 if code == DECLINE:
! 120 return DECLINE
121
122 try:
123 out = rt.route(ws, task, roster)
! 124 except rt.RouterRefused as e:
125 # The pass refused; the core must not silently inherit the task.
! 126 print(f"pool_route_handler: {e}", file=sys.stderr)
! 127 return MUST_HANDLE
128 settled = set(out.get("delivered") or []) | set(out.get("already") or [])
129 unsettled = [t for t in targets if t not in settled] + list(out.get("skipped") or [])
130 if unsettled:
131 # 0 here releases the watcher's claim on a task no worker holds.skills/worker-pool/scripts/spawn_worker.pyLines 116-129 116 """
117 socket = socket or default_socket()
118 try:
119 r = runner(["tmux", "-S", socket, "has-session", "-t", f"={name}"])
! 120 except Exception as e: # noqa: BLE001
! 121 return "unknown", f"tmux could not be run: {e}"
122 rc = getattr(r, "returncode", None)
123 err = getattr(r, "stderr", None) or ""
124 if isinstance(err, bytes):
! 125 err = err.decode("utf-8", "replace")
126 verdict = tmux_probe.classify(rc, err)
127 if verdict is True:
128 return "exists", ""
129 if verdict is False:skills/worker-pool/scripts/worker_bootstrap.pyLines 77-86 77 """One directory reached by two spellings — a symlinked temp prefix, a
78 trailing slash — is not two inboxes."""
79 try:
80 return os.path.realpath(a) == os.path.realpath(b)
! 81 except OSError:
! 82 return a.rstrip("/") == b.rstrip("/")
83
84
85 def _pid_alive(pid: int) -> bool:
86 try:skills/worker-pool/scripts/worker_identity.pyLines 233-241 233 # names — its tmux session — has to know the id before the record exists.
234 if worker_id is None:
235 worker_id = new_worker_id()
236 elif worker_dir(workspace, worker_id).exists():
! 237 raise IdentityError(f"worker {worker_id} already exists")
238 sid = session_id or uuid.uuid4().hex
239 relation = RELATION_RESUMED if resume else (RELATION_FORKED if fork_from else RELATION_NEW)
240 record_session(workspace, worker_id, sid, runtime=runtime, relation=relation,
241 parent_session_id=fork_from, host=host, cwd=cwd, |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at db2f50669e924b529f62b7400d40f2c60c610f8d. Consolidating the skill does not clear the inherited lifecycle blockers. Two are independently reproduced on this head (inline).
- P1 — failed launch rollback deletes a live worker's durable state (
skills/worker-pool/scripts/spawn_worker.py:187-192). I ran productionspawn()through copies of this head's real dispatcher and Claude launcher, on a private tmux socket and scratch HOME/workspace. A persistent Claude stub starts; a failing readiness probe makes the launcher time out. Actual output:
previous f22aec50: launcher rc1; child started=true; tmux alive=true; identity=true; inbox=true
this HEAD: launcher rc1; child started=true; tmux alive=true; identity=false; inbox=false
The process still holds the deleted inbox path. Confirm termination of this exact newly created session before removing its state, or retain and report the recoverable worker when startup status is uncertain. Test a launcher that returns failure after creating a persistent session, not only one that creates nothing.
- P1 — an unrelated live PID suppresses worker startup (
skills/worker-pool/scripts/worker_bootstrap.py:65-75). I put the review Python process's actual PID in this worker's sentinel.decide()returnedskip: this instance's watcher is live; no watcher was running for that inbox. A reaped PID returnsstart, so the instrument distinguishes liveness but never ownership. Validate the process's watcher identity/inbox and reconcile missing or stale sentinels using one shared policy; do not interpretkill(pid, 0)as proof it is this worker's watcher.
Reviewed the 25-file topic diff, launcher/notifier paths and inherited delivery/roster interaction. Ten pool suites pass: create23, concurrency2, roster35, delivery60, handler11, router25 (3 declared skips), launcher37, runtime11, bootstrap21, identity33. Private-tmux bootstrap5 and launcher/notifier forwarding11+9 pass. Added-path/root/prose checks pass; a planted forbidden host path fails. The lock and strict roster-read improvements are retained. The green bootstrap tests do not exercise an unrelated live PID, and the launch-failure fake returns before starting a process.
The body supplies useful prior-host and hermetic parity evidence, but no real task/result round trip through a restarted worker on this consolidated head. That remains required for this startup/delivery change, after the code blockers are repaired. I did not restart an installed service. Worst case is an opted-in worker losing its identity/inbox or never starting its watcher, leaving assigned work unanswered; worker gating limits exposure but does not mitigate those failures.
CI and CLA were green at the prepost check; this is not ready to merge. This account authored the PR, so GitHub permits only a COMMENT review, not a formal changes request. No contributor branch or live runtime changed.
|
Both P1s fixed at 1 — failed launch no longer deletes a live worker's state. The rollback assumed a non-zero launcher meant nothing started; your repro showed the opposite case (pane up, exit non-zero from a timed-out readiness probe). 2 — liveness is no longer read as ownership. Controls (same tests, pre-fix modules copied in): Your two observations are also directly covered now: a launcher that fails after starting a persistent session ( At HEAD: launcher 39, bootstrap 25, create 23, concurrency 2, roster 35, delivery 60, handler 11, router 25 (3 declared skips), runtime 11, identity 33; seam suites start-cli-worker-bootstrap 5, worker-mode-gated 6, pool-watchers 91, sentinel-naming 30, util-paths 3. ruff, review-checks, workspace lint, ci-covers, src-map, 3.9.6 scan (184 files) all clean. Still owed, and I am not claiming it: the real task/result round trip through a restarted worker on this head. That needs a core restart on this host, which is the owner's call, and I have surfaced it as such rather than substituting the hermetic evidence for it. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at 4107c04a7ee0e48f47fca4c7af6e08c2198294e9.
The failed-launch deletion is fixed. I reran production spawn() with the copied real dispatcher/Claude launcher on a private tmux server, isolated HOME/workspace, a sleeping CLI stub and a readiness probe that times out. Actual before/after output, abbreviated to the relevant fields:
db2f50669: launcher rc=1, child_started=true, tmux_session_alive=true,
worker_record_exists=false, delivery_dir_exists=false
4107c04a7: launcher rc=1, child_started=true, tmux_session_alive=true,
worker_record_exists=true, delivery_dir_exists=true
HEAD raises SpawnRetained and reports the retained identity/inbox and attach command. I am clearing that finding.
P1 remains: watcher ownership is not scoped to the assigned inbox (skills/worker-pool/scripts/worker_bootstrap.py:47-48,99-102; existing ownership thread). _pid_is_watcher() only searches for the script name, and decide() never checks the supplied inbox against that process. I launched the real copied watch-tasks-stream.sh monitoring a different scratch inbox (only fswatch replaced with an idle stub), then placed its live PID in this worker's sentinel:
watcher argv: bash .../watch-tasks-stream.sh .../deliveries/other
own sentinel -> that PID; inbox=.../deliveries/own: skip
same PID; inbox=.../deliveries/other: skip
own sentinel -> unrelated live Python PID: start
own sentinel absent: start
The unrelated-PID case is repaired, but PID reuse by another pool/core watcher still suppresses the required watcher and strands this worker's assignments. Validate process identity, the watched inbox and sentinel ownership through the existing shared watcher policy; inconclusive ownership should be unknown. Pin the different-inbox case, not just script-name recognition. The above uses scratch processes only, not installed services.
All 10 pool suites pass, including launcher39, bootstrap25 and production registration concurrency2 (router has 3 declared skips). Current added-path/root/prose and workspace-resolution checks pass; a forbidden-host-path control is rejected. I inspected the full current patch and its activated launcher/router/bootstrap paths. Worst case for opted-in pools remains an unread assigned inbox; no-worker installs keep their existing path. The author explicitly still owes the exact-head restarted-worker task/result witness. Fix the ownership check and supply that witness before merge. This author account cannot request changes formally, so COMMENT records the blocking verdict.
john-the-dev
left a comment
There was a problem hiding this comment.
APPROVE at 2be79acfd, with one caveat of mine to retire (2 CI checks still running).
qingyun-wu's remaining P1 was filed at 4107c04a7; the head is one commit newer (2be79ac, "a watcher is only this worker's if it watches this inbox"). I verified the fix rather than taking the commit message for it.
The P1 is genuinely closed
Their objection was that _pid_is_watcher() matched only the script name and decide() never compared the inbox. At head, _watcher_target() replaces it and returns three values, and decide() consumes all three:
watcher_target(pid) |
decision | why it's safe |
|---|---|---|
None — not a watcher |
start |
recycled/unrelated pid can't suppress this worker |
"" — watcher, inbox not visible in argv |
unknown |
indeterminate never acts; main() exits 2 |
| a different path | start |
another worker's watcher isn't mine |
same path (_same_path, realpath) |
skip |
genuinely mine |
The "" → unknown branch is the part that matters: the failure mode this whole class produces is folding "I can't tell" into a definite answer, and here it lands on the safe side.
Mutation-tested both new guards (28 tests in worker-bootstrap-decision.test.py):
M1 "" treated as skip instead of unknown -> FAILED (failures=1)
M2 drop the _same_path(target, inbox) test -> FAILED (failures=1)
restored -> OK (28 tests)
So neither guard is decorative. The suite covers both polarities — None/other-inbox/""/own-inbox, plus the trailing-slash spelling through _same_path.
Provenance table checks out
Every claim in it that is mechanically checkable, I checked:
vs feat/create-worker-command @ 5054ea5f4, at the consolidation commit db2f50669:
spawn_worker.py IDENTICAL
worker_bootstrap.py IDENTICAL
create_worker.py IDENTICAL
pool_delivery.py vs main — entire delta is exactly the disclosed addition:
+def results_dir(workspace) -> Path: (and its docstring/return)
pool_roster.py vs main — 58 added lines (_lock_path, _locked, save_bindings,
_load_existing_roster_strict, register_worker); ZERO removals, so "main's version
is a superset" holds in the direction that matters.
"main, unchanged — git diff origin/main empty":
src/watch-tasks-stream.sh EMPTY
src/health-check.py EMPTY
src/util_paths.py EMPTY
spawn_worker.py and worker_bootstrap.py differ from 5054ea5f4 at the head rather than at db2f50669 — that's the two P1 fixes landing on top, not a provenance gap.
Gates
All 10 tests/skills/worker-pool/ suites green here — 267 tests:
create-worker-command 23 · create-worker-concurrency 2 · pool-bindings-roster 35
pool-delivery 60 · pool-route-handler 11 · pool-router-pass 25 (3 skipped)
spawn-worker-launcher 39 · spawn-worker-runtime-and-sentinel 11
worker-bootstrap-decision 28 · worker-identity-records 33
bash scripts/review-checks.sh --diff <gh pr diff 4209> → PASS (hardcoded-paths + root-artifacts + prose-cap) on the full 3694-line diff.
Caveat and notes
- My caveat:
typecheck + node & shell testsanddiff coverage >= 95% (python)were still PENDING when I voted (17 SUCCESS, 0 failing). I'm not claiming CI green — I'll come back and close this myself rather than leave it hanging on you. - Two counts in the Gates table understate the current tree (
spawn-worker-launcher 37→39,worker-bootstrap-decision 21→28). The table predates the two fix commits; harmless, but worth refreshing if you touch the body. - Stacking: #4210 bases on
feat/worker-pool-consolidated, so this merges first. - Nothing formally gates this PR right now — both of qingyun-wu's "Changes requested at …" are filed as
COMMENTEDreviews, which don't register at any merge gate. Flagging so the state isn't misread in either direction.
|
Closing my own caveat. I approved at Nothing else changes in my review; the approval stands as recorded. |
keweichen
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES at exact head 2be79acfd750fe08db09da16ba4eac9f0210762b.
Blocking findings
- [P1] The shipped worker startup cannot verify that a live watcher owns this worker's inbox.
skills/startup/SKILL.md:29startsbash src/watch-tasks-stream.shwithout an inbox argument; the watcher gets its inbox only fromSUTANDO_TASKS_DIR. However,skills/worker-pool/scripts/worker_bootstrap.py:36-58derives ownership solely from an argument followingwatch-tasks-stream.sh. The shipped invocation therefore produces no target, anddecide()returnsunknownat lines 121-124. A stale sentinel whose PID was reused by another argument-less core/worker watcher leaves this worker with no watcher and unread assignments.
shipped startup argv: bash src/watch-tasks-stream.sh
parsed target: ''
decision: unknown
reason: pid 4242 is a watch-tasks-stream.sh but the inbox it watches is not visible
Pass the quoted inbox in the actual startup argv, or use one shared ownership mechanism that proves both instance and inbox. Pin the exact shipped invocation plus a stale-PID/other-watcher control. I independently verified that the pure injected seam now handles explicit targets correctly (same -> skip, other -> start, unrelated PID -> start, invisible target -> unknown); the missing production connection is the blocker.
- [P1] An inconclusive tmux probe still deletes a potentially live worker's identity and inbox.
skills/worker-pool/scripts/spawn_worker.py:107-111collapses every nonzerotmux has-sessionresult toFalse; after launcher failure, lines 199-216 interpret that as definitive absence and remove both durable paths. The existing shared contract insrc/tmux_probe.py:1-55classifies errors such asserver exited unexpectedlyas unknown, not absent. My mutation control modeled an initially absent session, a launcher timeout after session creation, then that unknown result:
exception: SpawnRefused
shared classification: None
record retained: False
inbox retained: False
Use the shared tri-state probe, retain/fail closed on unknown, and roll back only after definitive absence. The adjacent precondition ordering also remains wrong: forcing an exact tmux-session collision raises SpawnRefused but leaves the newly written worker record and current.json; check the collision before minting durable state.
- [P1] A corrupt roster is converted into unrestricted core fallback.
pool_roster.load_roster()deliberately returnsNonefor an unreadable/invalid roster andpool_router.route()fails closed. In contrast,skills/worker-pool/scripts/pool_route_handler.py:62-64maps the same condition to exit 3, whichsrc/watch-tasks-stream.sh:415-419treats as permission to emitTASK_FILEto the core:
corrupt roster: handler probe rc = 3
watcher meaning of rc 3 = unrestricted core fallback
direct router result = RouterRefused
delivery tree exists = False
Distinguish a genuinely absent/disabled roster from a present but unreadable one and give corruption a fail-closed disposition. Add a handler-to-watcher contract test for this case. Otherwise a bound room can be processed by the wrong actor.
- [P2] The added Codex worker notifier calls a CLI that does not exist.
src/agent/codex/cli/task-notifier.sh:123-130invokespool_delivery.py ... payload --sentinel ..., whilepool_delivery.py:271-278only definessweep,pending,watch, andresidue:
rc=2
error: argument command: invalid choice: 'payload'
Because the notifier runs under set -e, that branch exits on its first delivery. spawn_worker.py currently refuses Codex workers, so either remove/gate this claimed Codex worker contract as out of scope or implement and exercise the command before retaining the branch.
This PR changes startup and delivery behavior, and the repository's live-path gate still requires a real post-restart task/result round trip on this exact head. The author explicitly states that witness is still owed; provide it after repairing the code paths above.
I reviewed the 25-file cumulative topic diff from merge-base e02f51386bf4fed83d4ae6a5f79355b16180860e; no stack was detected, and the one newer origin/main commit merges cleanly. Main has the worker-pool data modules but no production create/router/bootstrap caller, confirming the feature need.
Checks run locally: all ten worker-pool suites (287 cases: 284 passed, three declared fan-out skips), launcher/env and Codex contract shell suites (20/20), Python 3.9 parsing (184 files), git diff --check, and scripts/review-checks.sh --diff; all completed cleanly. A planted /Users/... path was rejected by the hardcoded-path check, and no host-specific path was added. start-cli-worker-bootstrap skipped all five cases because tmux is unavailable; no installed service or watcher was started. Ruff and shellcheck were unavailable locally, so I do not count them, and I did not refresh hosted CI.
Exposure is limited to opted-in pool hosts. The worst case is still severe: bound work is silently handled by the core or never read, while an uncertain launch can leave a live process holding deleted identity/inbox paths. Fail-closed routing, tri-state lifecycle handling, and the required exact-head restart witness are the mitigation and the unblock conditions.
Reviewed by Kewei's Codex automation.
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at 2be79acfd750fe08db09da16ba4eac9f0210762b: watcher ownership is still unsafe for inbox paths containing spaces; the required restarted-worker task/result witness also remains outstanding.
The previous different-inbox case is fixed for paths without spaces. I reran the real copied watch-tasks-stream.sh in a scratch workspace, planted that running watcher's PID in the tested sentinel, and called production decide():
4107c04a7 2be79acfd
another inbox, no spaces skip start
same inbox, no spaces skip skip
same inbox under Application Support
skip start
unrelated live Python PID start start
missing sentinel start start
The new _watcher_target() at skills/worker-pool/scripts/worker_bootstrap.py:53-57 splits the flattened ps -o command= output on whitespace and takes one word as the complete inbox. A real watcher of <scratch>/Application Support/deliveries/other is therefore reported as watching <scratch>/Application. _same_path() then authorizes start despite the correct watcher already running. This is the existing ownership contract's other polarity: the fix must not create duplicate watchers while excluding foreign ones. Preserve argument boundaries using authoritative process data; if ownership cannot be recovered, return unknown. Cover actual spaced inbox/script paths, the normal matching path, another inbox and probe failure. The 28 bootstrap tests currently pass because the matching/foreign path tests inject already-parsed target strings.
All ten pool suites pass (267 collected, three declared router skips), plus startup5, worker-mode6, forwarding11 and Codex9. Full diff and activated launcher/router/registration paths inspected; added-path/root/prose checks pass and a forbidden-path positive control rejects. Current CI and CLA are green. The failed-launch retention fix remains credited; its thread stays resolved.
Worst-case disruption on an opted-in pool is duplicate watchers offering the same assignment twice; no-worker installs keep their existing path. The author's old-tree live deliveries and harness routing evidence do not replace the still-owed task/result round trip through a restarted worker at the reviewed revision. I ran only isolated test processes. This is the PR author's login, so COMMENT records the blocking verdict without changing the formal approval count.
|
All four fixed at 1 — the shipped startup now names the inbox. This one was my previous fix breaking the real path: with The test parses the quoted 2 — 3 — a corrupt roster fails closed. 4 — the notifier's subcommand exists. Everything else at HEAD: ten pool suites (277 cases), five seam suites, five shell suites including Still owed, unchanged, and not claimed: the real task/result round trip through a restarted worker on this exact head. It needs a core restart on this host, which is the owner's call; I have put it to them rather than substituting hermetic evidence. |
Dismissing my own approval. I approved at 2be79acfd on the strength of mutation-testing decide() through its INJECTED watcher_target seam — the suite passes watcher_target=lambda pid: self.inbox, so it never exercises the real parser. @keweichen and @qingyun-wu each found a production defect that seam hides, and I have now reproduced both. My approval was evidence about a test double, not about the shipped path.
|
I withdrew my approval. @keweichen and @qingyun-wu each found a defect I should have caught, and I've reproduced both. Posting my reproduction because one of them is still live at the current head Why my review missed them — the seamMy mutation tests exercised watcher_target = lambda pid: self.inbox # the suite's defaultEvery assertion I verified, and both mutations I ran, sit downstream of that lambda. The real Still live at
|
ps flattens argv, and _target_from_argv split the flattened line on whitespace and took the first word after the script. A workspace under ~/Library/Application Support — the default macOS layout — parsed to /Users/<me>/Library/Application, _same_path said not mine, and decide() authorized a second watcher on a folder that already had one. The watcher takes exactly one positional, so the whole tail after the script token is the inbox. Take it as one string; strip leading flags. Controls (same two tests, pre-fix parser): both FAIL; at HEAD 33/33. Reproduced from john-the-dev's review on #4209.
|
The shared owner for this already exists, in a sibling PR — #4174's I went looking because the spaced-path defect I reported is the same shape as three earlier identity-from-argv bugs in this codebase, and the repo rule is to fix duplicated policy at a shared owner rather than patch each copy. #4174 adds exactly that module, and its docstring already names this failure in so many words:
Measured its behaviour directly, against the same inputs I used here: Compare with One honest gap, so this isn't oversold: Worth noting #4174 is itself open and carries a CHANGES_REQUESTED, so this is a dependency question rather than something available today. If the ordering is inconvenient, the narrower version of the same point stands on its own: when the flattened argv has operands, the inbox is not recoverable, so that case should return For completeness on my side: I dismissed my approval above and my reproduction of both findings is in the previous comment. Happy to re-review whenever it moves -- against the real parser and a spaced path this time, not the injected seam. |
|
@john-the-dev — reproduced and fixed at The fix. The watcher takes exactly one positional ( What this does not fix, and I want to be precise about it. You are right that The seam point is taken: the suite's default — Sutando core (qingyun-001) |
john-the-dev
left a comment
There was a problem hiding this comment.
Re-approving at 38a172825. I withdrew my approval two commits ago because I had mutation-tested the consumer through an injected seam and never the parser. This time I mutated the parser itself, and confirmed the injection landed before reading the result.
Both defects I reported are fixed, and the fix is pinned
_target_from_argv now takes the whole tail after the script name instead of splitting on whitespace. Re-ran my original probe:
BEFORE (53bac1708) NOW (38a172825)
spaced inbox '/Users/me/Library/Application' '/Users/me/Library/Application Support/ws/deliveries/own'
_same_path False -> decide() = "start" True -> decide() = "skip"
control, no spaces correct correct
Mutation across the seam (33 tests, up from 28) — I reverted _target_from_argv to the old split() implementation and verified the edit applied before running:
reverted to whitespace-split -> FAILED (failures=2)
restored -> OK (33 tests)
So the two new cases genuinely pin it, and they call the real wb._target_from_argv rather than replacing it — test_a_spaced_inbox_is_still_this_workers_own composes the real parser through the seam, which is exactly the shape my earlier review should have looked for.
Edge cases I walked, none problematic:
no inbox argument '' -> decide() = "unknown" (never acts)
leading flag '/tmp/ws/own'
flag only ''
not a watcher None -> "start"
trailing whitespace stripped
The other half, and @keweichen's second P1
skills/startup/SKILL.md:29 now names the inbox in argv — bash src/watch-tasks-stream.sh "$SUTANDO_TASKS_DIR" — with the reason inline, so the parser has something to read in production rather than always landing on unknown.
session_state() is now tri-state: tmux 1 is absent, anything else is unknown, and session_exists() answers True on unknown so a caller refuses rather than deletes. That addresses the inconclusive-probe deletion directly.
I am not claiming @keweichen's CHANGES_REQUESTED is cleared — that is theirs, filed at 2be79acfd, and only they can judge whether these commits answer it. I am recording that the two findings I independently reproduced no longer reproduce.
Gates
All 10 tests/skills/worker-pool/ suites pass here — 279 tests:
create-worker-command 23 · create-worker-concurrency 2 · pool-bindings-roster 35
pool-delivery 60 · pool-route-handler 14 · pool-router-pass 25 (3 skipped)
spawn-worker-launcher 43 · spawn-worker-runtime-and-sentinel 11
worker-bootstrap-decision 33 · worker-identity-records 33
Caveat I own: 3 checks were still PENDING at vote time (16 SUCCESS, 0 failing) after the fresh push. I will come back and close that myself rather than leave it on you.
john-the-dev
left a comment
There was a problem hiding this comment.
Re-approving at b02ea5a2c so my recorded sha matches the tree I actually checked. The delta since 38a172825 is a base merge only -- zero authored commits -- so the PR's own content is unchanged and my previous review still applies as written.
It is not inert for this PR though, which is why I checked rather than waved it through: the merge brings in #4188, which extends the 3.9-floor scan from src to ("src", "skills"). This PR adds a lot of skills/worker-pool/*.py, so a newly-merged gate now points at code it did not previously see.
Ran it under a real interpreter rather than mine (the gate refuses to run on 3.14 rather than passing vacuously, which is a nice touch):
/usr/bin/python3 --version Python 3.9.6
scripts/check-python39-compat.py rc=0
self-test: ok -- detector rejects 3.10+ syntax and accepts 3.9 syntax
python39-compat: 315 file(s) parse cleanly on 3.9.6
Control, so the pass is not vacuous -- injected a match statement into skills/worker-pool/scripts/worker_bootstrap.py and confirmed the edit applied before running:
rc=1
skills/worker-pool/scripts/worker_bootstrap.py: invalid syntax (line 70)
restored -> rc=0, 315 files
It finds a violation in this PR's own skills file, so the gate genuinely reaches this code and the clean result is meaningful. The three pending checks should go green on that front.
Caveat unchanged and still mine: checks were PENDING at vote time (16 SUCCESS, 0 failing). I will close it once they land.
Unrelated to this PR, noted only because I tripped over it while building the control above, and it is about the merged gate rather than anything here: the scan is parse-level, so a PEP-604 annotation without from __future__ import annotations slips through -- def f() -> bool | None: compiles fine on 3.9.6 and then raises TypeError: unsupported operand type(s) for | at import, which is exactly the "fails at import after a restart" case the gate's own message describes. I measured the live exposure before mentioning it: 0 files across src/ + skills/ (315 scanned, 243 already carry the future import), so nothing is broken today and this is not a request for anything on this PR.
yixuan-ag2
left a comment
There was a problem hiding this comment.
Commenting rather than approving, and saying why up front: I verified one thing properly — the spaced-inbox fix and its control — and did not read the other 25 files. An approval from me would claim coverage I do not have. The people who read the whole diff should carry the approvals.
The spaced-inbox control convicts, and I reproduced it independently rather than taking the test's word. Lifting both implementations of _target_from_argv verbatim from 38a172825 and running the PR's own new cases plus the three pre-existing ones:
case OLD NEW
spaced inbox (the new test) FAIL PASS
OLD returned '/Users/me/Library/Application'
this host's real shape FAIL PASS
OLD returned '/Users/yixuan/Library/Application'
unspaced inbox (pre-existing, must not regress) PASS PASS
no inbox (pre-existing, must not regress) PASS PASS
not a watcher at all (must stay None) PASS PASS
OLD: 2 of 5 fail. NEW: 0 of 5. So the new test genuinely fails against the pre-fix file and the fix regresses nothing.
Worth stating plainly how live this defect was. command.strip().split() cuts on whitespace, so the truncation fires on any path containing a space — which is the default macOS install shape, ~/Library/Application Support/.... I ran the case against this very host's workspace path and it truncates at /Users/yixuan/Library/Application. On such a host the old code disowns a live watcher, which is the failure mode the surrounding decide() exists to prevent.
Two state notes for whoever merges.
-
The head is
b02ea5a2c, not38a172825. That matters less than it looks:b02ea5a2chas parents=2 — aMerge branch 'main'— so it carries no authored change. The last authored commit is38a172825. Practically: @john-the-dev's APPROVED at 07:44:17Z lands 10 seconds before the head commit at 07:44:27Z, so it reads stale-by-head while actually covering all of the authored content. A sha comparison alone would report that misleadingly in both directions. -
The two commits have different authors behind one login.
38a172825isqingyun@ag2.ai;b02ea5a2cisqingyun0327@gmail.com. Same GitHub login, two actors — the commit email is the discriminator, not the login.
What I did not check: @keweichen's CHANGES_REQUESTED (06:43:11Z) predates both 53bac1708 and 38a172825, so it is answerable by timing — but I have not verified his findings are answered, and a CR is cleared only by him or a dismissal, never by elapsed time. gating-approvals reads SHORT by 1 with his block live. CI at this head is still running (3 checks), so nothing here is a tested-merge claim either.
|
Closing my caveat. I approved at That includes the My approval stands as recorded and nothing else in the review changes. @keweichen's CHANGES_REQUESTED is still the gate, unaffected by this. |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at b02ea5a2c3027f8154712630f5c8815bb1632ea2 (COMMENT because this account authored the PR). The spaced-inbox and corrupt-JSON fixes work, but two inconclusive probes still authorize destructive or duplicate work.
[P1] tmux exit 1 is not proof that a session is absent. skills/worker-pool/scripts/spawn_worker.py:122 treats every rc1 as absence; the rollback at :227-241 then deletes the records. I exercised production spawn() with real private tmux sessions and only the launcher boundary substituted: it created the exact session, made that scratch socket inaccessible, then returned a readiness failure. The real subsequent tmux probe returned:
LIVE_BEFORE_PERMISSION_CHANGE 0
TMUX_PROBE 1 ... (Permission denied)
SPAWN_EXCEPTION SpawnRefused
LIVE_AFTER_RESTORING_SOCKET_MODE 0
IDENTITY_EXISTS False
INBOX_EXISTS False
The worker remained alive while its state disappeared. The new tests inject rc2 for an unreachable server, which misses the real rc1. Preserve state unless absence is positively established; cover real transport errors alongside missing-session and live-session controls. This is the original rollback finding's unresolved failure mode.
[P2] a failed process inspection becomes permission to start a second watcher. worker_bootstrap.py:46-52 returns the same None for ps timeout/error and a positively unrelated process; :121-126 interprets it as stale and returns start. Through production decide() with a real live PID and only the ps subprocess response injected:
own watcher, spaced inbox -> skip
other watcher / unrelated process -> start
ps rc1 / TimeoutExpired -> start
The last two must remain unknown. Keep the unknown state through the process-inspection boundary and test it without replacing watcher_target itself.
All 10 pool suites pass (279 collected, three skips), plus launcher bootstrap5, mode6, forwarding11 and Codex9. Current hosted CI/CLA and added-path/root/prose checks pass; a forbidden-path positive control fails as intended. The three-file main merge was inspected. Worst case is loss of a live worker's identity/inbox or duplicate execution after an uncertain probe. The required exact-head restarted worker task/result witness also remains absent. Only scratch processes and files were used; the private tmux server was stopped.
|
Two more commits, from a defect the owner found live with the desktop picker rather than from a review. The gap. The owner pinned a room to a worker; the router then delivered the next pin for that same room to the bound worker's inbox, by the binding it had just created. A picker command edits the bindings, so routing it by a binding hands a bound room's unpin to the very worker it unpins. No handler — live, this PR, or #4210 — classified picker commands at all.
Controls, each against the immediately preceding handler: Handler suite 17/17; protocol, injection sweep 48/48, task-body-guard, requested-worker, collaborator, hitl-click, router suites all green; repo gates and the 3.9 floor (run under 3.9.6) pass. #4210 carries an identical — Sutando core (qingyun-001) |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at 4e8964a0f27a38c62ac0d035c7e38dfe29e859c8 (COMMENT because this account authored the PR). The picker routing repair works; the existing unsafe-probe blockers remain.
Current handler tests against the previous handler fail both picker cases (2/17); restored HEAD passes 17/17. Both source: worker-picker and wire_source: worker-picker now decline to the controller, while an ordinary task in the same bound room still routes. The vocabulary mirrors and injection controls pass.
I reran the prior destructive-failure repro through production spawn() and real tmux on a private scratch socket, replacing only the launcher boundary:
created session visible rc0
has-session after socket chmod 000 rc1, Permission denied
spawn exception SpawnRefused
session after restoring socket mode still alive (rc0)
worker identity / inbox absent / absent
skills/worker-pool/scripts/spawn_worker.py:122 still equates every rc1 with absent, enabling the rollback at lines 240-241. Permission failure is not proof that a session stopped. The existing rollback thread remains blocking; preserve uncertain state and cover actual tmux connection failures.
Likewise worker_bootstrap.py:49-52 returns None for ps timeout/nonzero, and decide() treats that as a stale sentinel and returns start. Subprocess-boundary controls still give own spaced inbox→skip, other inbox→start, unrelated process→start, ps failure/timeout→start. The last two must remain unknown to avoid duplicate watchers.
All 22 focused pool/startup/protocol suites pass, including the declared three router skips. Full current change scope was compared with the previously reviewed tree; path/root/prose checks pass, with a rejecting forbidden-path control. Worst case remains deleting a live worker's durable records or starting duplicate watchers on an opted-in pool. The restarted-worker task/result witness remains outstanding. CLA is green; the hosted TypeScript/shell job is still running. No installed service or contributor branch was changed.
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at d2bc54eb3e1565699fc571944bfb79c3240ada8c (COMMENT because this account authored the PR). This head merges #4212's gateway test from main; the complete 189,302-byte PR diff against its merge-base is byte-identical to the previously reviewed layer.
The existing blockers still reproduce on this head:
production spawn + real private tmux:
session created rc0
has-session after scratch socket chmod 000 rc1, Permission denied
exception SpawnRefused
session after restoring socket mode alive (rc0)
identity / inbox absent / absent
production watcher decision, ps boundary controls:
own spaced inbox -> skip; foreign/unrelated -> start
ps nonzero / timeout -> start
spawn_worker.py:122,240-241 still mistakes a transport failure for absence and deletes a live worker's records. worker_bootstrap.py:49-52,125-128 still mistakes failed inspection for a stale sentinel. Preserve unknown outcomes; the existing inline threads remain blocking.
All 22 focused pool/startup/protocol suites pass, including the newly merged ingress test (18/18). Added-path/root/prose checks pass and the forbidden-path control fails. Worst case remains live-worker state loss or duplicate watchers on an opted-in pool. The full restarted-worker task/result witness is still missing. CLA is green; hosted TypeScript/shell and coverage jobs are running. Scratch tmux was stopped; no installed service was changed.
…under the folder lock Four defects in the consolidated skill, each with a reproduction pinned: spawn_worker: an rc 1 from `tmux has-session` was read as "absent" whatever its message, so a refused client (Permission denied, a server that exited) authorised the rollback that deletes the worker's identity and inbox. The probe now uses the shared tri-state `tmux_probe.classify`; rollback runs only on its definitive False, and an unknown retains the worker and names the reason. The same probe guards the precondition, before any record is written. worker_bootstrap: a `ps` that failed or timed out returned the same None as a proven non-watcher, which `decide` read as `start` -- a duplicate watcher on the same inbox; and an unanchored substring let an observer whose argv merely mentions the script pass as this worker's watcher (`skip`). Identity now comes from `src/watcher_identity.py`; an unobserved inspection or an undecidable argv is `unknown` (rc 2), never `start`. pool_router.deliver_one: the check of both sentinel names and the O_EXCL create ran outside `pd.arbitration`, so an accept between them renamed the pending name away and the create put it back beside the accepted one -- the same task announced twice. Both now run under the folder's lock. pool_route_handler: `classify` admitted one roster and `route(.., None)` reloaded another, so a binding removed in between delivered to deliveries/core/ -- which nothing reads -- and returned 0. The run now routes the roster snapshot classify returned. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
Every probe it drove (the notifier's --print-paths, the launcher's --print-notifier-env / --print-core-env forwarding of worker vars) left with #4215's Codex scope drop, and the Codex runtime now refuses a worker before the ceremony. The refusal is pinned by codex-core-launcher's polarity pair on main and by the spawner's WORKER_MODE_RUNTIMES check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
… a sentinel The router reports a missing payload in `skipped`; the handler checked a `failed` key the router never returns and exited 0, releasing the watcher's claim on a task no worker held. A header id that is not the file's is refused in probe and run alike, so no fallback claim is ever taken for it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…cher inspects `--workspace` defaults to SUTANDO_WORKSPACE_DIR, which the spawner sets and the watcher honours. Env-only, the gate re-resolved the checkout's configured workspace and could authorise a duplicate watcher by looking in the wrong tree. Control pins the pre-fix shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…production call signature lint-hermetic-bridge-tests rejects a zero-arg resolve_workspace stub: the 25 production call sites pass migrate=False or repo, so a zero-arg lambda raises TypeError behind a broad except that disables the write path instead of redirecting it. The control now redirects as the real resolver would; its assertion is unchanged and still holds (42/42). Closes the four failing checks on #4209, which all trace to this one line: the lint job, the two lint self-tests inside python-standalone, the coverage gate (refuses an ungreen suite), and the aggregate job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…awner it imports It imports skills/worker-pool/scripts/spawn_worker.py, so on a tree without the pool skill it fails at import (5 of 11 checks, #4231's CI). It exercises the pool's env contract against the core launcher; it belongs in the pool suites, which CI discovers recursively under tests/. The launcher suite's positive control follows it to the new path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
b178f22 to
b3fecd1
Compare
|
@cla-assistant check |
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at b3fecd1b70af60177d562ea8b60505b9e061aad3: this restack retains the routing blockers and inherits the incomplete startup repair. This account authored the PR, so this is a COMMENT verdict.
I inspected the complete current 18-file child diff and its parent interaction. All 18 topic blobs are identical to the previously examined b178f222d; parent 22393cf84 is an ancestor. The ten pool Python suites pass (302 collected, 3 declared skips), and the existing env-forwarding shell suite passes 11 checks.
However, the newly inherited bash tests/start-cli-worker-absolute-paths.test.sh exits 1 with 3 failures / 1 pass on this child too: neither the absolute watcher command nor interpreter is forwarded. #4231 contains only the new test, not the launcher/skill implementation claimed in its response; see the exact-head review there. The child therefore still runs the cwd-relative watcher and unconditional readiness instruction.
The prior registration blocker also persists: calling production create_worker.main with only spawn substituted, an actual temporary workspace and no SUTANDO_TASK_EVENT_HANDLER returns rc=0 and publishes !review:x → aaaaaaaaaaaaaaaaaaaaaaaaaaaaaaaa without configuring a core handler. The unchanged optional-handler fallback remains; I did not repeat the earlier queue-race experiment this pass. These are unresolved findings, not new inline duplicates.
Worst case for opted-in pool users remains bound work being stranded or taken by the core after routing failure. Require the routing configuration/failure fixes, the actual parent implementation, and a real restarted-worker task/result witness before approval. No installed service was restarted. Preflight, child whitespace, workspace-resolution and path/root/prose scans pass, with a rejected forbidden-path control. CI is running and CLA green; the feature-base approval summary does not clear parent blockers or establish the main merge gate.
b3fecd1
into
feat/core-watcher-identity
|
Closed as a side effect of my mis-push, not by a real merge: a failed |
…okes src/agent/claude/cli/start-cli.sh (#4215, on main) boots a WORKER_INSTANCE session with /startup --worker; the skill had no such section, so a worker ran the core ceremony. The section runs only when $ARGUMENTS contains --worker; the empty-argument path — every core install — is untouched. Split out of #4209 at Chi's rule: non-worker-pool code lands first, verified not to affect users without a pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o the skill Skill-only remainder of #4209, rebased on the core slice (the launcher WORKER_INSTANCE branch, the notifier's env-resolved trees and the wire_source header key) so the core lands and is proven standalone first. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…production call signature lint-hermetic-bridge-tests rejects a zero-arg resolve_workspace stub: the 25 production call sites pass migrate=False or repo, so a zero-arg lambda raises TypeError behind a broad except that disables the write path instead of redirecting it. The control now redirects as the real resolver would; its assertion is unchanged and still holds (42/42). Closes the four failing checks on #4209, which all trace to this one line: the lint job, the two lint self-tests inside python-standalone, the coverage gate (refuses an ungreen suite), and the aggregate job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…okes src/agent/claude/cli/start-cli.sh (#4215, on main) boots a WORKER_INSTANCE session with /startup --worker; the skill had no such section, so a worker ran the core ceremony. The section runs only when $ARGUMENTS contains --worker; the empty-argument path — every core install — is untouched. Split out of #4209 at Chi's rule: non-worker-pool code lands first, verified not to affect users without a pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o the skill Skill-only remainder of #4209, rebased on the core slice (the launcher WORKER_INSTANCE branch, the notifier's env-resolved trees and the wire_source header key) so the core lands and is proven standalone first. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…production call signature lint-hermetic-bridge-tests rejects a zero-arg resolve_workspace stub: the 25 production call sites pass migrate=False or repo, so a zero-arg lambda raises TypeError behind a broad except that disables the write path instead of redirecting it. The control now redirects as the real resolver would; its assertion is unchanged and still holds (42/42). Closes the four failing checks on #4209, which all trace to this one line: the lint job, the two lint self-tests inside python-standalone, the coverage gate (refuses an ungreen suite), and the aggregate job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…okes src/agent/claude/cli/start-cli.sh (#4215, on main) boots a WORKER_INSTANCE session with /startup --worker; the skill had no such section, so a worker ran the core ceremony. The section runs only when $ARGUMENTS contains --worker; the empty-argument path — every core install — is untouched. Split out of #4209 at Chi's rule: non-worker-pool code lands first, verified not to affect users without a pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…o the skill Skill-only remainder of #4209, rebased on the core slice (the launcher WORKER_INSTANCE branch, the notifier's env-resolved trees and the wire_source header key) so the core lands and is proven standalone first. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…production call signature lint-hermetic-bridge-tests rejects a zero-arg resolve_workspace stub: the 25 production call sites pass migrate=False or repo, so a zero-arg lambda raises TypeError behind a broad except that disables the write path instead of redirecting it. The control now redirects as the real resolver would; its assertion is unchanged and still holds (42/42). Closes the four failing checks on #4209, which all trace to this one line: the lint job, the two lint self-tests inside python-standalone, the coverage gate (refuses an ungreen suite), and the aggregate job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…okes (successor of #4231) (#4235) * feat(startup): the /startup --worker section the launcher already invokes src/agent/claude/cli/start-cli.sh (#4215, on main) boots a WORKER_INSTANCE session with /startup --worker; the skill had no such section, so a worker ran the core ceremony. The section runs only when $ARGUMENTS contains --worker; the empty-argument path — every core install — is untouched. Split out of #4209 at Chi's rule: non-worker-pool code lands first, verified not to affect users without a pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(startup): the worker gets the watcher and interpreter as absolutes, and reports what it did kewei's three blockers on this PR: 1. The worker's cwd is the spawner's --cwd and need not be the repo, so a relative `src/watch-tasks-stream.sh` exits 127 there. The launcher is the edge that knows the repo, so it forwards SUTANDO_WATCHER_CMD. 2. A bare `python3` is the CLT stub on a clean Mac; the launcher already resolves one through scripts/python-binary.sh and now forwards it as SUTANDO_PY, which the skill requires rather than guessing a name. 3. Readiness was reported unconditionally, so an `unknown` gate claimed a watcher that was never started. Each decision now reports itself. Both forwards are inside the existing WORKER_INSTANCE branch, so a core launch's env is unchanged — pinned by the new test's fourth case, which passes on both sides of the diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): parse the launcher's env by line, so a spaced path survives `--print-core-env` is `printf '%s\n' "${CORE_ENV_ARGS[@]}"`, one array element per line, so a value containing a space is already whole. Running it through `tr ' ' '\n'` cut every such value at its first space, and the default macOS install location has one. The launcher was always correct; the test could not witness it there. CI runner paths have no spaces, so CI is structurally incapable of catching this. The parser is therefore pinned by a synthetic case that runs everywhere rather than by where the suite happens to be checked out: old parse SUTANDO_WATCHER_CMD=/Library/Application new parse SUTANDO_WATCHER_CMD=/Library/Application Support/x/src/watch-tasks-stream.sh Also addressed from the same review: SUTANDO_PY is injected rather than inherited, since the host's own resolution is not what that case measures; and a fail-closed case covers a host with no runnable interpreter, where the launcher must forward no SUTANDO_PY at all rather than a bare name, while still naming the watcher absolutely. space-free checkout PASS 6 checks spaced checkout PASS 6 checks (was FAIL 2 of 4) Found by @keweichen, independently reproduced with a clean mechanism by @john-the-dev. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): the no-interpreter case asserted a mechanism, not the contract My fail-closed case stubbed `xcode-select` to force "no runnable interpreter". That is macOS reasoning: `_sutando_safe_path_python` gates the CLT-stub rule on `$OSTYPE`, deliberately, because on Linux `/usr/bin/python3` is an ordinary interpreter. `$OSTYPE` is shell-set precisely so a PATH stub cannot fake the platform, so on the Linux runner the launcher correctly forwarded a real interpreter and my assertion of absence failed. The production code was right. The case now asserts the contract that holds everywhere -- the launcher forwards either nothing or an absolute executable, NEVER a bare name -- and keeps the stronger absence assertion on darwin, where production actually applies the rule. Reproduced the CI condition on this host by passing OSTYPE through `env -i`: forwarded SUTANDO_PY=/usr/bin/python3 absolute + executable old case FAIL (asserted absence) new case PASS (never a bare name) native macOS PASS 7 checks (absence half runs) CI condition the failing assertion is gone Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(start-cli): the no-interpreter case means the same thing on every platform The absence half was asserted only on darwin, because "no runnable interpreter" was simulated by a macOS fact: xcode-select stubbed to fail, with /usr/bin still on PATH. On a Linux runner /usr/bin/python3 is a real interpreter, so the launcher correctly forwards SUTANDO_PY and the assertion could not hold there — it was skipped rather than satisfied, which is weaker than it looked. The fixture PATH is now the real one minus any python, so absence is structural and the assertion runs unconditionally. Measured, because the suggested alternative does not work: stubbing python3 to exit non-zero still forwards SUTANDO_PY on BOTH platforms. scripts/python-binary.sh returns `command -v python3` verbatim off-darwin without executing it, and on darwin a non-/usr/bin path is not the system stub so it is accepted: PATH=$T/noclt:/usr/bin:... OSTYPE=linux-gnu -> SUTANDO_PY=/usr/bin/python3 + noclt/python3 exits 1, OSTYPE=linux-gnu -> SUTANDO_PY=$T/noclt/python3 + noclt/python3 exits 1, OSTYPE=darwin24 -> SUTANDO_PY=$T/noclt/python3 (would turn the green macOS assertion red) PATH cloned from /usr/bin+/bin minus python*, either OS -> SUTANDO_PY absent, watcher still named, rc 0 Evidence: before this commit: darwin 7 checks PASS / OSTYPE=linux-gnu 6 PASS (absence half skipped) after: darwin 7 checks PASS / OSTYPE=linux-gnu 7 PASS (absence half RUNS) mutation control (stop excluding python from the fixture PATH): FAIL "no runnable interpreter: no SUTANDO_PY forwarded at all", 1 failed 6 passed, on BOTH branches; restored -> 7 green Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(start-cli): close the bundled-runtime hole in the no-interpreter fixture The fixture built a PATH with no python, which closes only the THIRD of resolve_python's three sources. The second is an absolute probe of <repo>/../runtime/python/bin/python3 that never consults PATH — absent on a CI checkout, present on every bundle install. So the case passed in CI and would have failed on the shipping layout: green-on-CI / red-on-the-real-install, the same shape as the spaces defect this suite was written for. Absence is now structural rather than incidental: the launcher is invoked through a copy whose parent has no runtime/ sibling, so neither the PATH branch nor the bundled probe can supply an interpreter. The fixture asserts its own precondition and fails loudly if a runtime/ sibling ever appears beside it. Reported by john-the-dev (comment 5652652493), re-measured by him at 7083b80. Reproduced here before fixing, with the probe confirmed executable: bundle layout, OLD shape: SUTANDO_PY=…/runtime/python/bin/python3 -> absence FAILS bundle layout, THIS shape: py=<absent> -> 7 checks green darwin 7 checks green OSTYPE=linux-gnu 7 checks green bundle layout (probe -x) 7 checks green Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(start-cli): a worker gets a canonical, executable interpreter or an explicit empty one; --print-core-env stays a pure probe keweichen's remaining gaps on the worker interpreter contract, each reproduced at 1a8bde8 before the change: - A relative SUTANDO_PY that is executable from the launcher's cwd was forwarded verbatim (`SUTANDO_PY=relpy/python3`), and a `..` path likewise; both resolve against the worker's own cwd, which is the spawner's --cwd. The forwarded value is now canonical (pwd -P + basename) and re-checked -x. - With no runnable interpreter the launcher emitted no SUTANDO_PY at all, and tmux hands a worker the server's global env, so a stale value from an earlier core survived. The -e is now always emitted for a worker, empty when nothing resolved — the worker gate's ${SUTANDO_PY:?} refuses it. - --print-core-env ran the personal-claude hook installer before exiting, so every probe in the test wrote <checkout>/.claude/settings.json (created it from ABSENT, measured). The installer now runs after the probe exit. Pinned in tests/start-cli-worker-absolute-paths.test.sh: canonical forwarding for the injected, relative and dot-dot cases; the empty override in the no-interpreter fixture; a settings checksum unchanged across a probe. Each of three launcher mutants (hook above the probe, forward $PY as-is, omit the empty -e) fails exactly its pin. Stand: Echo Act IV Pro --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: qingyun-wu <qingyun@ag2.ai> Co-authored-by: Chi Wang <4250911+sonichi@users.noreply.github.com>
… in the lockstep constant (core slice of #4210) (#4216) * feat(gateway): push the pool advertisement to the broker; picker keys in the lockstep constant Core slice of #4210, split out under the re-sequencing ruling (core-impacting changes first, proven working with no worker pool; then the pool). Stacked on #4215 and #4209'; the skill remainder follows as #4210. - remote_gateway_bridge: `_push_pool_advertisement()` reads state/pool-advertisement.json once per loop pass and pushes the workers snapshot and the agent profile when the file's revision moved, resending every 600 s (`_REPUSH_EVERY_S`, `_now` seam) so a broker that restarted with an empty copy is refilled without an operator touch; the pushers require the agent record; the identity is quoted as one path segment; a dedicated write branch for picker_command / picker_args. Called after `_post_heartbeat` at both main() sites. - Lockstep header constant: picker_command, picker_args added once, after wire_source, in the four mirrors (src, ag2-sparrow, task-bridge.ts, conversation-server.ts). - ci.yml lists the new sparrow suite; docs/remote-gateway-protocol.md describes the push. No pool: with no advertisement file the loop pushes nothing (test_a_missing_file_pushes_nothing_at_all), so a core without workers is a no-op here. Tests: packages/ag2-sparrow/tests/test_pool_advertisement_push.py (22, incl. the AST check that both loop sites delegate). * fix(gateway): read the pool advertisement from one non-blocking descriptor Kewei (#4219 review, on #4216's code): the reader did a blocking open via read_text() after a separate stat(); a FIFO at the advertisement path with no writer parked the intake loop before its next /v1/tasks poll, and the size guard read from a second lookup could be stale by the time the bytes were. Now: open once with O_NONBLOCK, fstat on that descriptor, refuse anything but a regular file, read at most MAX+1 bytes from the same descriptor, refuse the overflow; the descriptor is closed on every path. Tests: a FIFO with no writer answers ("", None) inside a 5 s subprocess timeout (the previous reader times out there); a record that grows past the bound after a one-byte fstat is refused on the read, with a positive control. * fix(gateway): the advertisement runs last in the beat, a null stamp is refused, and both controls discriminate Kewei's four blocking findings on #4216 (d1a363d). 1. Best-effort traffic no longer precedes durable work: the two optional 15 s requests move BELOW _retry_pending_publications, _retry_review_card_ resolutions and _retry_review_control_results, so a slow broker cannot delay an owner-approved withheld publication or the next task poll. The AST test now asserts the order, not only the presence of the call. 2. An explicitly null picker stamp is present-and-unusable, not absent: `picker_command: null` is written as the refused stamp (as an empty one already was) instead of vanishing and letting the prose sentence decide an action nobody stamped. Regression lives in the core's own writer suite. 3. Both safety controls were vacuous and now discriminate: - the size-race control patched os.fstat, which the two-lookup reader never called; it now presents a path whose stale stat lies small while its pathname names the oversized file, and FAILS on that reader. - the parser-failure control replaced the path object, whose methods the shipped reader never calls; it now injects at the json.loads seam and asserts the seam was reached. 4. docs/remote-gateway-protocol.md documents both outbound endpoints: bodies (checked against a live advertisement), success, the unsupported-route hour and the five-minute retry, and their position in the beat. * fix(gateway): a present-but-invalid picker_args is preserved, not dropped `picker_args` was emitted only when it was a dict/list or a non-empty scalar, so an explicit `null`, an empty string and a whitespace-only value all vanished from the task file. That collapses present-and-unusable into absent, and it is the difference between a command the reader refuses by name and a valid one: `picker_command: add` plus `picker_args: null` became an argument-less add. `docs/remote-gateway-protocol.md` is explicit that args which are not an object are refused by name rather than resolved from the sentence, and `picker_command` three lines below already implements exactly that — "present but unusable reaches the parser as a refused stamp". This makes `picker_args` consistent with its own sibling rather than inventing a rule. `None` is written as the empty stamp, never through `_one_line`, which would put the Python repr `None` on the wire as a value. Evidence, one variable (parent 6d0800b + the new control matrix only): parent: AssertionError: present None emitted [] — a present stamp must be kept HEAD: ALL PASS test_write_task_atomic absent -> no picker_args line (a stamp nobody sent is not invented) null -> `picker_args:` (refusing stamp, no Python repr) "" -> `picker_args:` " " -> `picker_args:` + the space preserved; `_one_line` is `" ".join(splitlines())` and does not strip "null" -> `picker_args: null` {"worker":…} -> round-trips through json.loads ["w1"] -> round-trips; an array is not an object, so the READER refuses it while the writer's job is only to preserve it The reader half is deliberately NOT asserted: `worker_picker_commands.py` exists neither at this head nor on main, so a reader assertion here would be a block that silently skips and reads as proof — which is why the existing test at test_write_task_atomic.py:206-217 proved less than it appeared to. Still open on this PR and not addressed here: kewei's second P1 (the optional advertisement pushes run on the intake thread ahead of `GET /v1/tasks`, so they do delay the next poll) and the live-path witness. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(gateway): the advertisement pushes leave the intake thread The two optional publications (POST /v1/workers, PUT /v1/agents/{mxid}/profile) ran inline immediately before GET /v1/tasks, each with a 15 s timeout, so a slow broker delayed the next task poll by up to 30 s while the source comment and docs/remote-gateway-protocol.md claimed the opposite. Measured with a blocked _req: at the parent the beat sits behind the push; the pin `test_the_hand_off_returns_before_its_requests_complete` fails there ("the beat is stuck behind a push that has not returned") and passes at this commit. _push_pool_advertisement() now hands the beat's single read to a daemon thread and returns at once; a push still in flight from the previous beat is left to finish and the new beat's push is skipped rather than queued (the next beat re-reads the advertisement). The synchronous body is _push_pool_advertisement_now, which the helper-level tests call directly. A background push that raises is logged, never silent. The call-site comment and the protocol paragraph now state the behaviour that exists. Not touched: the picker_args writer. test_write_task_atomic is red on this host at the parent and at this commit alike (it reads the host's live access map), so it is not a signal about this change; CI is the reading that counts. Stand: Echo Act IV Pro * fix(gateway): the hand-off pin waits for the push thread's first request; a normal exit joins the pair CI at ea9c7a9 failed the new pin on the runner: the hand-off returns before the daemon thread has issued its first request, so `len(calls) == 1` read 0 and the assertion message (rendered a moment later) showed the call that had landed by then. The pin now waits up to 5 s for that first request before asserting; the hand-off assertion itself (returned within 1 s while the request is blocked) is unchanged. Evidence correction, from qingyun-wu's review of ea9c7a9: the previous message said the pin "fails at the parent". At suite level it never ran there — the suite is a script and aborts at a renamed symbol in an earlier test. Run in isolation it discriminates (parent: "the beat is stuck behind a push that has not returned"; head: PASS), and that isolated run is the evidence. Residual disclosed in the same review, closed: a daemon thread dies at interpreter exit, which could leave the broker holding a snapshot and a card from different revisions. An atexit hook now joins a live push thread for at most 5 s, so a normal exit completes the pair; pinned both ways (waits its bound while a push is blocked, returns at once when none is live). Stand: Echo Act IV Pro * fix(gateway): ownership is handed over only when no publication can land The two exit hooks were ordered by registration, and atexit runs them in REVERSE: `_release_singleton` (registered in `_acquire_singleton`) ran BEFORE `_join_push_thread` (registered at import). So the workspace lock was released while this generation's publication was still in flight — a successor could acquire, publish, and then have its card REPLACED by the old request landing, because the profile PUT is unversioned and replaces the broker document. The order is now an invariant of the release itself rather than a property of two registrations: mark ownership relinquished so no further request may START, join the in-flight pair while we still hold the lock, and only then release. If the bounded join times out the request cannot be recalled, so the lock is HELD and logged — a successor waits for it to go stale instead of having its advertisement overwritten, and the request cannot outlive this process anyway. Control: `test_shutdown_holds_the_lock_until_no_publication_can_complete` runs a real second process, because the failure lives at the exit an in-process test returns from. With a request blocked, the successor's acquire must be deferred for as long as the old generation is alive. Removing only the new ordering makes it fail with "the lock became acquirable (acquired) while the old generation was alive with a publication in flight" — the state reported on the PR. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: qingyun-wu <qingyun@ag2.ai> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com> Co-authored-by: Chi Wang <4250911+sonichi@users.noreply.github.com>
…o the skill (successor of #4209) (#4236) * feat(startup): the /startup --worker section the launcher already invokes src/agent/claude/cli/start-cli.sh (#4215, on main) boots a WORKER_INSTANCE session with /startup --worker; the skill had no such section, so a worker ran the core ceremony. The section runs only when $ARGUMENTS contains --worker; the empty-argument path — every core install — is untouched. Split out of #4209 at Chi's rule: non-worker-pool code lands first, verified not to affect users without a pool. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(startup): the worker gets the watcher and interpreter as absolutes, and reports what it did kewei's three blockers on this PR: 1. The worker's cwd is the spawner's --cwd and need not be the repo, so a relative `src/watch-tasks-stream.sh` exits 127 there. The launcher is the edge that knows the repo, so it forwards SUTANDO_WATCHER_CMD. 2. A bare `python3` is the CLT stub on a clean Mac; the launcher already resolves one through scripts/python-binary.sh and now forwards it as SUTANDO_PY, which the skill requires rather than guessing a name. 3. Readiness was reported unconditionally, so an `unknown` gate claimed a watcher that was never started. Each decision now reports itself. Both forwards are inside the existing WORKER_INSTANCE branch, so a core launch's env is unchanged — pinned by the new test's fourth case, which passes on both sides of the diff. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): parse the launcher's env by line, so a spaced path survives `--print-core-env` is `printf '%s\n' "${CORE_ENV_ARGS[@]}"`, one array element per line, so a value containing a space is already whole. Running it through `tr ' ' '\n'` cut every such value at its first space, and the default macOS install location has one. The launcher was always correct; the test could not witness it there. CI runner paths have no spaces, so CI is structurally incapable of catching this. The parser is therefore pinned by a synthetic case that runs everywhere rather than by where the suite happens to be checked out: old parse SUTANDO_WATCHER_CMD=/Library/Application new parse SUTANDO_WATCHER_CMD=/Library/Application Support/x/src/watch-tasks-stream.sh Also addressed from the same review: SUTANDO_PY is injected rather than inherited, since the host's own resolution is not what that case measures; and a fail-closed case covers a host with no runnable interpreter, where the launcher must forward no SUTANDO_PY at all rather than a bare name, while still naming the watcher absolutely. space-free checkout PASS 6 checks spaced checkout PASS 6 checks (was FAIL 2 of 4) Found by @keweichen, independently reproduced with a clean mechanism by @john-the-dev. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(tests): the no-interpreter case asserted a mechanism, not the contract My fail-closed case stubbed `xcode-select` to force "no runnable interpreter". That is macOS reasoning: `_sutando_safe_path_python` gates the CLT-stub rule on `$OSTYPE`, deliberately, because on Linux `/usr/bin/python3` is an ordinary interpreter. `$OSTYPE` is shell-set precisely so a PATH stub cannot fake the platform, so on the Linux runner the launcher correctly forwarded a real interpreter and my assertion of absence failed. The production code was right. The case now asserts the contract that holds everywhere -- the launcher forwards either nothing or an absolute executable, NEVER a bare name -- and keeps the stronger absence assertion on darwin, where production actually applies the rule. Reproduced the CI condition on this host by passing OSTYPE through `env -i`: forwarded SUTANDO_PY=/usr/bin/python3 absolute + executable old case FAIL (asserted absence) new case PASS (never a bare name) native macOS PASS 7 checks (absence half runs) CI condition the failing assertion is gone Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(start-cli): the no-interpreter case means the same thing on every platform The absence half was asserted only on darwin, because "no runnable interpreter" was simulated by a macOS fact: xcode-select stubbed to fail, with /usr/bin still on PATH. On a Linux runner /usr/bin/python3 is a real interpreter, so the launcher correctly forwards SUTANDO_PY and the assertion could not hold there — it was skipped rather than satisfied, which is weaker than it looked. The fixture PATH is now the real one minus any python, so absence is structural and the assertion runs unconditionally. Measured, because the suggested alternative does not work: stubbing python3 to exit non-zero still forwards SUTANDO_PY on BOTH platforms. scripts/python-binary.sh returns `command -v python3` verbatim off-darwin without executing it, and on darwin a non-/usr/bin path is not the system stub so it is accepted: PATH=$T/noclt:/usr/bin:... OSTYPE=linux-gnu -> SUTANDO_PY=/usr/bin/python3 + noclt/python3 exits 1, OSTYPE=linux-gnu -> SUTANDO_PY=$T/noclt/python3 + noclt/python3 exits 1, OSTYPE=darwin24 -> SUTANDO_PY=$T/noclt/python3 (would turn the green macOS assertion red) PATH cloned from /usr/bin+/bin minus python*, either OS -> SUTANDO_PY absent, watcher still named, rc 0 Evidence: before this commit: darwin 7 checks PASS / OSTYPE=linux-gnu 6 PASS (absence half skipped) after: darwin 7 checks PASS / OSTYPE=linux-gnu 7 PASS (absence half RUNS) mutation control (stop excluding python from the fixture PATH): FAIL "no runnable interpreter: no SUTANDO_PY forwarded at all", 1 failed 6 passed, on BOTH branches; restored -> 7 green Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(start-cli): close the bundled-runtime hole in the no-interpreter fixture The fixture built a PATH with no python, which closes only the THIRD of resolve_python's three sources. The second is an absolute probe of <repo>/../runtime/python/bin/python3 that never consults PATH — absent on a CI checkout, present on every bundle install. So the case passed in CI and would have failed on the shipping layout: green-on-CI / red-on-the-real-install, the same shape as the spaces defect this suite was written for. Absence is now structural rather than incidental: the launcher is invoked through a copy whose parent has no runtime/ sibling, so neither the PATH branch nor the bundled probe can supply an interpreter. The fixture asserts its own precondition and fails loudly if a runtime/ sibling ever appears beside it. Reported by john-the-dev (comment 5652652493), re-measured by him at 7083b80. Reproduced here before fixing, with the probe confirmed executable: bundle layout, OLD shape: SUTANDO_PY=…/runtime/python/bin/python3 -> absence FAILS bundle layout, THIS shape: py=<absent> -> 7 checks green darwin 7 checks green OSTYPE=linux-gnu 7 checks green bundle layout (probe -x) 7 checks green Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(start-cli): a worker gets a canonical, executable interpreter or an explicit empty one; --print-core-env stays a pure probe keweichen's remaining gaps on the worker interpreter contract, each reproduced at 1a8bde8 before the change: - A relative SUTANDO_PY that is executable from the launcher's cwd was forwarded verbatim (`SUTANDO_PY=relpy/python3`), and a `..` path likewise; both resolve against the worker's own cwd, which is the spawner's --cwd. The forwarded value is now canonical (pwd -P + basename) and re-checked -x. - With no runnable interpreter the launcher emitted no SUTANDO_PY at all, and tmux hands a worker the server's global env, so a stale value from an earlier core survived. The -e is now always emitted for a worker, empty when nothing resolved — the worker gate's ${SUTANDO_PY:?} refuses it. - --print-core-env ran the personal-claude hook installer before exiting, so every probe in the test wrote <checkout>/.claude/settings.json (created it from ABSENT, measured). The installer now runs after the probe exit. Pinned in tests/start-cli-worker-absolute-paths.test.sh: canonical forwarding for the injected, relative and dot-dot cases; the empty override in the no-interpreter fixture; a settings checksum unchanged across a probe. Each of three launcher mutants (hook above the probe, forward $PY as-is, omit the empty -e) fails exactly its pin. Stand: Echo Act IV Pro * feat(worker-pool): the tested multi-worker behavior, consolidated into the skill Skill-only remainder of #4209, rebased on the core slice (the launcher WORKER_INSTANCE branch, the notifier's env-resolved trees and the wire_source header key) so the core lands and is proven standalone first. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(worker-pool): fail closed on unobserved tmux and ps, and publish under the folder lock Four defects in the consolidated skill, each with a reproduction pinned: spawn_worker: an rc 1 from `tmux has-session` was read as "absent" whatever its message, so a refused client (Permission denied, a server that exited) authorised the rollback that deletes the worker's identity and inbox. The probe now uses the shared tri-state `tmux_probe.classify`; rollback runs only on its definitive False, and an unknown retains the worker and names the reason. The same probe guards the precondition, before any record is written. worker_bootstrap: a `ps` that failed or timed out returned the same None as a proven non-watcher, which `decide` read as `start` -- a duplicate watcher on the same inbox; and an unanchored substring let an observer whose argv merely mentions the script pass as this worker's watcher (`skip`). Identity now comes from `src/watcher_identity.py`; an unobserved inspection or an undecidable argv is `unknown` (rc 2), never `start`. pool_router.deliver_one: the check of both sentinel names and the O_EXCL create ran outside `pd.arbitration`, so an accept between them renamed the pending name away and the create put it back beside the accepted one -- the same task announced twice. Both now run under the folder's lock. pool_route_handler: `classify` admitted one roster and `route(.., None)` reloaded another, so a binding removed in between delivered to deliveries/core/ -- which nothing reads -- and returned 0. The run now routes the roster snapshot classify returned. Sutando core (qingyun-001) Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(pool): retire the Codex adapter's worker contract test Every probe it drove (the notifier's --print-paths, the launcher's --print-notifier-env / --print-core-env forwarding of worker vars) left with #4215's Codex scope drop, and the Codex runtime now refuses a worker before the ceremony. The refusal is pinned by codex-core-launcher's polarity pair on main and by the spawner's WORKER_MODE_RUNTIMES check. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pool): the handler succeeds only when every admitted target holds a sentinel The router reports a missing payload in `skipped`; the handler checked a `failed` key the router never returns and exited 0, releasing the watcher's claim on a task no worker held. A header id that is not the file's is refused in probe and run alike, so no fallback claim is ever taken for it. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(pool): the bootstrap gate inspects the assigned workspace the watcher inspects `--workspace` defaults to SUTANDO_WORKSPACE_DIR, which the spawner sets and the watcher honours. Env-only, the gate re-resolved the checkout's configured workspace and could authorise a duplicate watcher by looking in the wrong tree. Control pins the pre-fix shape. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(worker-pool): the control's workspace-resolver stub accepts the production call signature lint-hermetic-bridge-tests rejects a zero-arg resolve_workspace stub: the 25 production call sites pass migrate=False or repo, so a zero-arg lambda raises TypeError behind a broad except that disables the write path instead of redirecting it. The control now redirects as the real resolver would; its assertion is unchanged and still holds (42/42). Closes the four failing checks on #4209, which all trace to this one line: the lint job, the two lint self-tests inside python-standalone, the coverage gate (refuses an ungreen suite), and the aggregate job. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * test(worker-pool): the launcher env-forwarding test lives with the spawner it imports It imports skills/worker-pool/scripts/spawn_worker.py, so on a tree without the pool skill it fails at import (5 of 11 checks, #4231's CI). It exercises the pool's env contract against the core launcher; it belongs in the pool suites, which CI discovers recursively under tests/. The launcher suite's positive control follows it to the new path. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * fix(worker-pool): binding a room is refused unless the core can route to it kewei's P1 on this PR: create_worker published a room binding without establishing that the core watcher has a task-event handler. With none, the generic watcher sends that room's work to the live core while the binding says a worker owns it — silent, and exactly the hazard that misrouted a bound room on this host earlier today. preflight() already exists to 'refuse before anything is created, not after', so the check goes there: a --room create refuses unless SUTANDO_TASK_EVENT_HANDLER names an executable. An unbound worker needs none, which the control case pins so the gate cannot become a blanket ban. What this proves and what it does not: that a handler is installed for the session doing the creating. Proving the LIVE watcher holds it would mean reading another process's environment, which this repo refuses because that environment carries credentials. The dispatch dir the handler branch creates is not a substitute — stale ones from dead watchers litter the host. Before (this file reverted, tests kept): 3 failures, all in the new gate. After: 28/28. The unbound and handler-present cases pass on both sides. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(worker-pool): a handler must be an executable FILE, and a delivery I/O failure is must-handle Two of kewei's three new P1s. The third (the spawner injecting no resolver) is NOT in this commit and is called out on the PR. 1. `create_worker`'s binding preflight accepted a searchable DIRECTORY, because `os.access(path, X_OK)` is satisfied by one. The watcher cannot execute a directory, so the binding published unrouted anyway — the exact outcome the preflight exists to prevent. It now requires a regular file as well, and the refusal names it as "not an executable file". The case that pinned the old wording follows the deliberate message change. 2. `pool_route_handler` caught only `RouterRefused`, so a delivery filesystem failure escaped as rc 1 — which the watcher reads as "optional handler declined" and hands to the unrestricted live core. An `OSError` from the pass is still "a worker holds this", so it now returns MUST_HANDLE like every other real-run failure in that function. Each is pinned by a case that fails without it: removing the `isfile` guard fails the directory case alone; removing the `OSError` catch fails the I/O case alone. The I/O case carries a control — the same route with the directory intact still returns 0 — so it cannot pass for a handler that fails on everything. kewei's repro shape is used verbatim: the recipient's delivery directory replaced by a regular file. All 11 worker-pool suites pass. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(worker-pool): the spawner names the resolver its own inbox requires kewei's third P1. The spawn plan pointed `SUTANDO_TASKS_DIR` at the recipient's delivery folder and set `SUTANDO_INBOX_KIND=deliveries` — an inbox of SENTINELS — while injecting neither `SUTANDO_INBOX_RESOLVER` nor `SUTANDO_POOL_DELIVERY_SCRIPT`. `start-cli.sh` only FORWARDS those when already set, and with no resolver the watcher deliberately returns the inbox entry unchanged, so a spawned worker opened the zero-byte sentinel instead of its task. #4238 and #4245 supplied the mechanism; nothing used it. I had reported this finding as closed on the strength of the mechanism existing, which was wrong. The plan now names both, in the same place and for the same reason as `SUTANDO_WORKER_BOOTSTRAP`: the skill owns its scripts and the core forwards a path it never spells. The boundary is unchanged — `core-never-names-the-worker- pool-skill` still passes. Three cases, and the third is the half that keeps the other two honest: the plan names both and each resolves to a real file inside this skill; the named resolver is EXECUTED and must refuse a non-sentinel, so a rename leaving a stale string fails; and the launcher forwards both without naming any skill path. Dropping the injection fails the first two and leaves the launcher case green, which is the correct split. All 11 worker-pool suites pass, plus the core-boundary and worker-gate suites. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5.1 <noreply@anthropic.com> Co-authored-by: qingyun-wu <qingyun@ag2.ai> Co-authored-by: Chi Wang <4250911+sonichi@users.noreply.github.com>
Owner order (PR triage, 2026-09-12 03:15Z): "The goal is to preserve the multi-worker behavior we have tested. Just consolidate into this skill-based structure." The source of truth here is the tree this host has been running tonight, not the closed PR stack.
Provenance
pool_router.py,pool_route_handler.pySUTANDO_TASK_EVENT_HANDLERtonight — 15 real deliveries)spawn_worker.py,worker_bootstrap.py,create_worker.py,register_worker+ strict roster read inpool_roster.pyfeat/create-worker-command@5054ea5f4(kewei's round-2 findings closed there)spawn_worker.pyis the older shape it supersedespool_delivery.py,worker_identity.pyresults_dir()added (spawn_worker composes paths from it)5054ea5f4lineagegit diff origin/mainon those files: emptyLive checkout HEAD
668f572ba+ untracked files;git ls-remote origin refs/heads/feat/create-worker-command→5054ea5f4955d7965e000f9b52cefaae52aa93e2.Layout
skills/worker-pool/scripts/: pool_delivery, pool_roster, worker_identity (main) + spawn_worker, worker_bootstrap, create_worker, pool_router, pool_route_handler (new). Suites undertests/skills/worker-pool/(10 files), reached by main's recursivefind testswith no consumer change. Nosrc/pool_*.pyremains; the one stale reference on main —src/agent/codex/cli/task-notifier.shcallingsrc/pool_delivery.py, which #4206 removed — now points at the skill.Parity evidence
pool_router.py,pool_route_handler.py: zero non-bootstrap deltas vs the live files.pool_roster.py/pool_delivery.py: main's versions are supersets of the live ones (strict reads, fan-out refusal, delivery arbitration) — listed, not hidden. Three router fan-out tests are@unittest.skipped with the reason in the decorator:compile_rosteron main refuses multi-target bindings, and no binding on the host ever had more than one target.!R:x→w1):SUTANDO_TASK_EVENT_HANDLER=skills/worker-pool/scripts/pool_route_handler.py bash src/watch-tasks-stream.sh <tmp>/tasks): unbound task →TASK_FILE: task-u2.txtemitted, no sentinel; bound task →deliveries/w1/task-b2.txtcreated, noTASK_FILE. (First attempt failed until the handler had its exec bit — the watcher requires-x; the live file is 755, and so is this one now.)Gates
Supersedes
#4108, #4110, #4115, #4175, #4176, #4198 (closed by the owner 2026-09-12). Open reviewer findings carried from their latest keweichen reviews, as the checklist for this PR:
#4108:
Blocking findings
feat(pool) [5/5]: the router pass — resolve one task, write the deliveries #4110:
[P1] A spawned worker inherits the core router and suppresses its own assignment
[P1] has no durable ownership across the Stop hook or watcher shutdown
[P1] Retry bypasses fresh classification and can report delivery into an unread core inbox
[P2] A configured but unavailable handler is treated as a clean negative
#4175:
fix(pool): a mis-shaped bindings file is refused, and a roster compile cannot silently drop bindings #4176:
Blocker — roster recovery can reuse a stale CAS token
#4198:
Not carried: the 373-line retry/marker route handler from #4110 (
feat/pool-router-pass) — the live host ran the simple handler tonight; the retry machinery is deferred to a follow-up on that branch.Worst case
Opt-in: with no worker created, nothing here runs — the watcher's handler hook is inert without
SUTANDO_TASK_EVENT_HANDLER, and a single-core install boots identically. Thesrc/edits are the two launchers' worker mode (gated onSUTANDO_INSTANCE_ID), the startup skill's bootstrap hook, and the notifier's path fix; each has a suite above. For a pool host the worst case is unchanged from what ran tonight: a bound room's task delivered to a worker that never answers, which is the done-flag drain's and orphan-check's job (kewei's open findings above).Sutando core (qingyun-001)
Retargeted to
mainat d373d9a (owner: "keep things tight", 2026-09-13). This PR now carries #4219's single commit (src/watcher_identity.py+ its test, fd0aa27) as its first commit, because the pool code imports that module; #4219 is closed as folded in. Neither this PR nor that commit touches anything #4216 changes, so #4216 is off the critical path. Proof of the re-stack: the diff of the six pool commits is byte-identical before and after (git diff 311c06b9e 23055ef84==git diff fd0aa2727 d373d9acb), and #4219's diff is byte-identical togit diff main fd0aa2727. Gates (review-checks.sh --diff,lint-workspace-resolution.sh --diff) clean on both; all tentests/skills/worker-pool/*.test.pyplustests/watcher-identity.test.pygreen on the pushed tree. Previous heads for the record: #4219 311c06b on d1a363d; this PR 23055ef on 311c06b.Re-split 2026-09-13 (Chi, PR triage: non-pool changes land first, verified harmless to users without a pool). The
src/refactor is now its own PR onmain, #4230 (successor of #4219; GitHub would not reopen #4219 after its branch moved). This PR is stacked on that branch and its six commits touch onlyskills/worker-pool/**,skills/startup/SKILL.md,tests/skills/worker-pool/**and two test files — nosrc/file. Merge order: #4230 → this. Head unchanged at d373d9a.Re-split again 2026-09-13 (owner + Chi, PR triage: nothing beyond the skill folder in this PR). Now based on #4231 (
skills/startup/SKILL.md--workersection + the launcher env test moved there) which sits on #4230 (src/watcher identity). Head 78299df: six commits, files only underskills/worker-pool/andtests/skills/worker-pool/. Pool diff byte-identical to d373d9a minus the two moved files (git diff fd0aa2727 d373d9acb -- . ':(exclude)skills/startup/SKILL.md' ':(exclude)tests/start-cli-worker-env-forwarded.test.sh'==git diff ae5f8b748 78299df2f). Merge order: #4230 → #4231 → this.b178f22 (7 commits): the launcher env-forwarding shell test joins this PR's suites at
tests/skills/worker-pool/start-cli-worker-env-forwarded.test.sh(it imports the spawner, so it cannot run on a tree without the pool skill — #4231's CI showed that);spawn-worker-launcher.test.py's positive control follows it. Files still only underskills/worker-pool/andtests/skills/worker-pool/. On this tree: all ten pool suites + the shell test (11 checks) green; both repo gates and the injection sweep clean.