Repository navigation
docs(worker-pool): document bind_room's missing task-event-handler publish - #4828
Conversation
…blish bind_room() (pin-an-existing-worker-to-a-new-room) never calls publish_task_event_handler(), unlike register_worker()/create_worker() which call it on every registration. A room pinned only through bind_room() on an install that never registered a worker the other way silently never gets a task-event handler declared, so bound tasks fall through to the core instead of the worker -- no error, the room still gets an answer, just from the wrong instance. Documents how to check (task_event_handler_config_path(...).exists()) and the safe stopgap (call publish_task_event_handler() directly -- the exact thing register_worker() would have done). The actual code fix (bind_room() calling it itself) is tracked in #4580, still open. Found and manually applied on a live host after the gap caused silently misrouted DMs; the owner asked for it to be documented so the same diagnostic path doesn't need rediscovering.
|
Independently re-verified every claim in this review before filing (not taking it on trust):
Approving on that basis — docs-only, technically accurate, diff itself is 0-blocking. Two non-blocking nits carried over from the original review (add Per this repo's own merge rule, this needs the failing CI re-run or fixed on main before merge regardless of review state — flagging so it isn't merged on this approval alone while red. (Posting as a plain comment, not a formal review: this PR is authored under |
john-the-dev
left a comment
There was a problem hiding this comment.
Approve at 462f223. Docs-only (skills/worker-pool/SKILL.md, +49/−0). I checked every claim against source at this head:
pool_roster.py:362register_worker()callspublish_task_event_handler(workspace)(line 379), before the roster/binding writes. Verified.pool_roster.py:395bind_room()doessave_bindings+compile_rosterand never callspublish_task_event_handler. Verified, so a room pinned on an install with noregister_worker()history never getsstate/task-event-handler.json.util_paths.task_event_handler_config_path(state_dir)returnsstate_dir/"task-event-handler.json"(src/util_paths.py:586). The check snippet resolves the same file the writer uses. Verified.- The watcher reads that exact path via
util_paths.py task-event-handler-config-path(src/watch-tasks-stream.sh:326) and re-reads it per event (:376). That backs the "no restart" claim. Verified. - #4580 is open ("declare the router on every path that makes a pool exist"). Verified.
Nits (non-blocking):
- Both snippets assume cwd = repo root (
sys.path.insert(0, 'src')/'skills/worker-pool/scripts') and$WSalready set. One line saying so would help.
CI: the red tests/watch-tasks-stream-fallback-receipt-scope.test.py (run 36339311484, python 4/5) cannot be reached by a SKILL.md-only diff. main is green at ef09093 and c0fa67b. I did not rerun it here, so it is unproven rather than author-caused; a rerun or update-branch should clear it.
…w section Per Susan's bot's review of #4828: the two bash snippets used $WS without defining it (a bare unset variable silently produces a wrong path and a false "False"), and both assume cwd is the repo root for their sys.path inserts without saying so in prose. Both fixed.
|
@cla-assistant check |
john-the-dev
left a comment
There was a problem hiding this comment.
Re-approve at 121c528. The only change since my approval at 462f223 is +4/−2 in skills/worker-pool/SKILL.md: both snippets now define WS via scripts/sutando-config.sh workspace, and the prose says to run them from the repo root. That is exactly the nit I raised; nothing else changed (compare 462f223...121c528: 1 commit, 1 file).
yixuan-ag2
left a comment
There was a problem hiding this comment.
APPROVE at 121c528314. john-the-dev has already verified every source claim here — register_worker() publishing at pool_roster.py:379, bind_room() never doing so, and task_event_handler_config_path resolving the same file the watcher re-reads per event — so I am not re-filing any of that. I confirmed the two I looked at independently and agree; the rest of this is only what his review does not cover.
I ran the check snippet on a live install, and the answer is the interesting one
A doc whose value is "check with this command" is worthless if the command errors, so verbatim from the repo root:
python3 -c "import sys; sys.path.insert(0, 'src'); from util_paths import task_event_handler_config_path; print(task_event_handler_config_path('$WS/state').exists())"
False
False on a real host. Following your own next step to see whether that matters:
roster.json -> {"workers": {"656b3bf…": {"state":"live"}}, "bindings": {}}
bindings.json ABSENT
task-event-handler.json ABSENT
deliveries/656b3bf…/ present
One live worker, zero bindings. Your failure mode needs a pinned room — "every room pinned since then silently falls through to the core" — so nothing is misrouted here; the host is primed for the bug rather than suffering it. I deliberately did not run the stopgap: with no binding it would publish a handler that routes nothing, and "safe to run any time" is not a reason to change live routing state for no benefit.
That is the one thing I would add to the doc, in a clause: a reader who runs the check, sees False, and applies the stopgap unconditionally is acting on a pool with no bindings, where there is nothing to repair. bindings.json absent (or roster.json's bindings empty) is the cheap discriminator between "primed" and "broken", and it decides whether the stopgap is the fix or a no-op. Non-blocking.
I also like that the last paragraph refuses to oversell itself — "a one-time fix for this host's current state, not a code fix … tracked in #4580". That is what most stopgap docs get wrong.
CI is red and none of it is yours
python standalone tests (1/5) 18:21:26->18:27:03 FAILURE tests/codex-core-launcher.test.py
python standalone tests (2/5) 18:21:26->18:25:02 FAILURE tests/health-check-skills-driver-drift.test.py
tsc + tests (clean install) 18:27:05->18:27:09 FAILURE
A docs-only diff cannot plausibly reach any of these, but I wanted better than "cannot plausibly":
tsc + tests (clean install)is not an independent failure — it is therequired-testsjob (needs: […python-standalone-tests]) and it ran 4 seconds, starting 2 seconds after 1/5 finished. It is echoing the two above it.- 1/5 is
codex-core-launcher.test.py,FileNotFoundError: /tmp/…/tmux.log— the #4729 signature. It also fails atorigin/mainlocally, but by the other arm (AssertionError: managed core-input monitor did not start, #4431). So "known-broken in two ways", not "fails identically" — both issues open. - 2/5 I could not dismiss by inspection, so I ran it:
health-check-skills-driver-drift.test.pypasses atorigin/mainlocally. The discriminator is the detail string rather than the verdict — CI gotHEAD in the skills checkout is unknown — not asserting drift, locally it gotcould not identify the driver stamp 9b5b85b (ambiguous prefix, unreadable object …). Two different branches of the same probe, so the test is sensitive to whether the runner has a skills checkout HEAD. Environment, not your diff.
Not a change request, and not something reviewing can fix — flagged only because mergeStateStatus is BLOCKED and a reader could think this PR introduced it.
What I did not do: three tests and two snippets, not the suite; and I did not exercise a bound room end to end, so the fall-through itself rests on your account plus the source, not on my having reproduced it.
Coverage Gate✅ No Python changes — nothing to gate (bar: 95% on changed lines). |
bassilkhilo-ag2
left a comment
There was a problem hiding this comment.
Post-merge verification at head 121c528 (merged as dabbbe6). Posted as COMMENT because the PR is already merged with two approvals on this head. Findings are for a follow-up, not a revert.
Core claim verified (correct):
skills/worker-pool/scripts/pool_roster.py:395-413bind_room()doessave_bindingsthencompile_roster, with nopublish_task_event_handlercall.pool_roster.py:379:register_worker()callspublish_task_event_handler(workspace)before any roster or binding write.create_worker.py:105goes throughregister_worker.compile_roster(270-308) and_publish(311-318) only write the roster and advertisement, not the handler. The "nothing declares the handler" claim holds.publish_task_event_handler(340-360) writestask_event_handler_config_path(<ws>/state)=state/task-event-handler.json(src/util_paths.py:586-591) pointing atpool_route_handler.py. The stopgap snippet makes the same call.- The "no restart" claim holds:
src/watch-tasks-stream.sh:320-340resolves that path and fswatches its directory. deliveries/<worker id>/sentinels matchpool_delivery.py:5-9,85-86.- The picker pin path is real:
worker_picker_commands.py:381callspr.bind_room.
Doc inaccuracies (nits, follow-up):
- "
pool_roster.py bindboth go through it":pool_roster.pyhas no CLI (no__main__, argparse orsys.argv), and nothing in the tree invokespool_roster.py bind. The only production caller is the picker atworker_picker_commands.py:381. Drop the CLI mention or name the picker path. - "a fresh install whose only registrations ever went through
bind_room()" cannot happen in-tree.bind_roomrefuses with "no roster; nothing to bind to" (403-405) when no roster exists. Every other roster writer (rename_worker,apply_profile_label_overrides,unbind_room) also requires an existing roster. The only writer that creates a roster,register_worker, publishes the handler first. The reachable cases are (a) the handler file is lost or deleted, and (b) a roster written by an engine from beforeregister_workerpublished the handler, when the launcher defaulted it (#4432). Suggest rewording the trigger list to those two. #4580 is still the right code fix.
CI: the final head is all green, but CI run 36340326297 passed only on attempt 2. Attempt 1 failed tests/codex-core-launcher.test.py and tests/health-check-skills-driver-drift.test.py. The red test named in the thread, tests/watch-tasks-stream-fallback-receipt-scope.test.py, failed on the earlier head 462f223 (run 36339311484, 5/6) and passed at 121c528. It is not red on main: I found no main failure of it in the last 30 CI runs, and main's CI on c0fa67b and ef09093 passed. Main's recent reds are other python tests failing once on a commit (claude-task-notifier-inflight on 194f2a5, lint-old-collab-literals on cb99b4e, watch-tasks-stream-handler-terminal-rc on f9f8a5a), which fits a flaky suite. "Unrelated" still holds on scope: the diff is skills/worker-pool/SKILL.md +52/-0 only, and that test does not reference SKILL.md or worker-pool. More precisely, it was a flake, not a red on main.
…and a hash-prefix collision tests/codex-core-launcher.test.py: several tests read the fake tmux's call log immediately after SIGTERM'ing the notifier process. On a slow runner the log file (created lazily on first invocation) can still not exist, turning a plain assertion failure into an unrelated FileNotFoundError. Most call sites already guarded this with `self.log.read_text() if self.log.exists() else ""`; one didn't. Extracted the guard into a `_tmux_calls()` helper and used it at every call site so the fix can't drift back out of sync again. tests/health-check-skills-driver-drift.test.py: case o)'s fixture relies on the stamped commit's object-dir prefix (first two hex chars) differing from the new HEAD's, so making the stamp's directory unreadable doesn't collaterally block resolving HEAD too. Git commit hashes are non-deterministic across runs, so this occasionally collides (~1/256), which drives check_skills_driver_code_drift() down the "HEAD is unknown" branch instead of the "stamp is INCONCLUSIVE" branch the test asserts on -- exactly the CI failure observed on #4828. Regenerate the second commit until its prefix genuinely differs from the stamp's, bounded at 20 attempts. Neither failure is caused by #4828's SKILL.md-only diff; both are pre-existing main-branch flakiness that happened to surface on its CI run. Stand: Echo Act IV Pro
…and a hash-prefix collision (#4834) tests/codex-core-launcher.test.py: several tests read the fake tmux's call log immediately after SIGTERM'ing the notifier process. On a slow runner the log file (created lazily on first invocation) can still not exist, turning a plain assertion failure into an unrelated FileNotFoundError. Most call sites already guarded this with `self.log.read_text() if self.log.exists() else ""`; one didn't. Extracted the guard into a `_tmux_calls()` helper and used it at every call site so the fix can't drift back out of sync again. tests/health-check-skills-driver-drift.test.py: case o)'s fixture relies on the stamped commit's object-dir prefix (first two hex chars) differing from the new HEAD's, so making the stamp's directory unreadable doesn't collaterally block resolving HEAD too. Git commit hashes are non-deterministic across runs, so this occasionally collides (~1/256), which drives check_skills_driver_code_drift() down the "HEAD is unknown" branch instead of the "stamp is INCONCLUSIVE" branch the test asserts on -- exactly the CI failure observed on #4828. Regenerate the second commit until its prefix genuinely differs from the stamp's, bounded at 20 attempts. Neither failure is caused by #4828's SKILL.md-only diff; both are pre-existing main-branch flakiness that happened to surface on its CI run. Stand: Echo Act IV Pro
Summary
bind_room()(pin-an-existing-worker-to-a-new-room) never callspublish_task_event_handler(), unlikeregister_worker()/create_worker(), which call it on every registration. On a host whose only registrations ever went throughbind_room(),state/task-event-handler.jsonnever gets written — so every room pinned there silently falls through to the core instead of the worker. No error; the room still gets an answer, just from the wrong instance.This was found live: worker-pool routing on a real host had been silently broken since 2026-09-25, discovered only because a DM to the core happened to differ from what the pinned worker would have said. The actual code gap is #4580, still open. This PR documents the diagnostic + a safe stopgap (calling
publish_task_event_handler()directly, the exact callregister_worker()makes) so the same investigation doesn't need rediscovering on the next host that hits it.Test plan
Docs-only change, no code touched.
🤖 Generated with Claude Code