Skip to content

fix(hooks): check-pending-tasks.sh understands worker-pool delegation - #4339

Merged
sonichi merged 7 commits into
mainfrom
fix/stop-hook-worker-pool-awareness
Sep 18, 2026
Merged

sonichi merged 7 commits into
mainfrom
fix/stop-hook-worker-pool-awareness

Conversation

@sonichi

@sonichi sonichi commented Sep 16, 2026

Copy link
Copy Markdown
Owner

Summary

  • Stop hook's only readiness signal was "does results/<id>.txt exist for tasks/<id>.txt". With SUTANDO_TASK_EVENT_HANDLER=pool_route_handler.py wired into a core's task watcher, a delegated task's payload stays in tasks/ by design (the worker reads it from there via the inbox resolver) — only a sentinel differs, in deliveries/<worker>/. The hook had no visibility into that, so the core kept blocking Stop on work a worker had already correctly taken over.
  • Symmetric bug on the worker side: a worker session inherits the same hook, which checked the core's tasks/ queue — so a worker got blocked by the core's own unrelated pending work.
  • Fix: core mode now skips any tasks/<id>.txt that has a live sentinel (pending .txt or accepted .accepted/.claimed) in some worker's own deliveries/ folder. Worker mode (SUTANDO_INSTANCE_ID set) judges only its own deliveries/<id>/ folder instead of the core's tasks/, reading the shared payload back from tasks/ for the block message text since the sentinel itself is a zero-byte marker. Block reason text is now mode-aware (Unprocessed tasks in tasks/ vs Unprocessed deliveries in deliveries/<id>/).

Fixes #4338, fixes #4281.

Before/after evidence

Both issues (#4338, #4281) describe the live symptom directly — reproduced live on this host today after wiring SUTANDO_TASK_EVENT_HANDLER for the first time: every room-pinned task correctly routed to its worker, but the core's Stop hook re-blocked on "Unprocessed tasks in tasks/" repeatedly until the worker finished (observed waits up to several minutes for a real PR-scoping task).

Before (this branch's parent, origin/main): no test exercised delegation at all — the gap was invisible to CI.

After:

$ bash tests/check-pending-tasks-workspace.test.sh
  ok   workspace task blocks
  ok   block payload names the task
  ok   result present suppresses the block
  ok   legacy <repo>/tasks/ does not block
  ok   empty queue emits {}
  ok   a refused interpreter is not worked around
  ok   a refused interpreter still emits valid JSON
PASS

$ bash tests/check-pending-tasks-worker-pool.test.sh
  ok   core does not block on a task delegated to a worker
  ok   core does not block on an accepted delegation either
  ok   core still blocks on a genuinely unclaimed task
  ok   worker is not blocked by the core's unrelated tasks/ queue
  ok   worker blocks on its own unresolved sentinel
  ok   worker's block payload names the task, read from the shared payload
  ok   worker's block reason names its own deliveries folder, not tasks/
  ok   worker's block clears once its result is ready
PASS

git diff --cached | bash scripts/review-checks.sh -> PARTIAL (hardcoded-paths + root-artifacts clean; prose-cap had no in-scope files) (expected -- this PR touches .sh files only, no .py).

Test plan

  • Existing tests/check-pending-tasks-workspace.test.sh -- 7/7, unchanged behavior for the non-worker-pool case
  • New tests/check-pending-tasks-worker-pool.test.sh -- 8/8, covers: core skips a pending-sentinel delegation, core skips an accepted-sentinel delegation, core still blocks a genuinely unclaimed task (control), worker ignores the core's unrelated queue, worker blocks on its own live sentinel, worker's block message names the task (read from the shared payload) and its own deliveries folder, worker's block clears once its result is ready
  • scripts/review-checks.sh clean

Generated with Claude Code

The Stop hook's only readiness signal was "does results/<id>.txt exist for
tasks/<id>.txt". Since SUTANDO_TASK_EVENT_HANDLER=pool_route_handler.py
delegates a task by writing a sentinel into deliveries/<worker>/ while the
payload stays in tasks/ by design, the core kept blocking Stop on work a
worker had already correctly taken over (#4338), and a worker session
inherited the same check against the core's unrelated tasks/ queue (#4281).

Now: core mode skips any tasks/<id>.txt with a live sentinel (pending or
accepted) in some worker's own deliveries/ folder — it's the router's
recipient, not a core orphan. Worker mode (SUTANDO_INSTANCE_ID set) judges
only its own deliveries/<id>/ folder instead of the core's tasks/, reading
the shared payload back from tasks/ for the block message since the
sentinel itself is a zero-byte marker.

New test: tests/check-pending-tasks-worker-pool.test.sh, 8 assertions
covering both directions plus the unclaimed-task control. Existing
tests/check-pending-tasks-workspace.test.sh unchanged, still 7/7.

Fixes #4338, fixes #4281.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@github-actions

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ No Python changes — nothing to gate (bar: 95% on changed lines).

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at 825837b2c. No blocking objection — holding the approval only until the job that runs your new test reports. Two non-blocking notes below.

The bug is real, and the diff proves it structurally. The removed loop was tasks/*.txt against results/<same>.txt and nothing else — there was no deliveries/ concept anywhere in the hook, so a delegated task could not be distinguished from an orphan. That is not a judgement call about severity; the old code had no input that could tell them apart. I also hit this hook's blocking path twice today from the other side (an unrecorded turn), so the "core keeps blocking on work it correctly handed away" shape is familiar.

The split reads correct. Core mode asking "is this still mine to report?" and worker mode asking "do I still owe a reply?" — answered from its own folder, never the core's queue — is the right decomposition, and extracting is_ready_result() keeps readiness owned by delivery/readiness.py instead of drifting. The three-suffix sentinel_task_id() matches what pool_delivery.py actually produces. The test covers both directions plus a genuinely-unclaimed control, which is the part that makes it a proof rather than a happy path.

Note 1 — worker mode can list one task twice (cosmetic)

The worker loop globs *.txt, *.accepted and *.claimed separately and appends per sentinel, with no dedup on the derived TASK_ID. Normally harmless, because pool_delivery.py renames .txt → .accepted rather than adding a file — but that same module explicitly handles "a stray pending name beside the accepted one" (pool_delivery.py:310), so the coexisting state is one it expects to see. In it, the Stop message lists the task twice.

Your new test models the rename (rm -f …$PROBE.txt then create .accepted), so it never enters that state. Worth a dedup on TASK_ID, or a comment saying double-listing is accepted.

Note 2 — a question, not an objection: what makes the unaccepted skip safe?

Core mode skips on any sentinel including an unaccepted one, and your comment states that intent plainly. The exposure, if nothing else moved, is that a worker dying between sentinel and result turns a blocking gate into a silent drop — the core stops reporting, and nothing else is watching.

I went looking before raising it and found the mitigation: pool_delivery.py has a stale-sentinel state and a released action, plus --stage abandon. So the answer is probably "the sweep reclaims it." Does that sweep run automatically on a stock install, or only when invoked? If it's manual-only, this trades a noisy gate for a quiet one; if it's on a timer, the skip is correct and I'd say so in the comment so the next reader doesn't re-derive it.

Why not APPROVE yet

14 checks green, CLA green, mergeable. Three are still pending, and one of them is typecheck + node & shell tests — the job that actually executes tests/check-pending-tasks-worker-pool.test.sh. Approving now would be approving the test's existence rather than its result. Ping me when it lands green and I'll convert this to an approval at the exact head; nothing in the code is holding me.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Review at 825837b2 (core seat of this login). Ran it rather than read it:

  • The new test measures the fix. tests/check-pending-tasks-worker-pool.test.sh on the extracted head tree: PASS. The same test file against the parent tree (13e7eb2f, this branch's base): FAIL — the core still emits {"decision":"block","reason":"Unprocessed tasks in tasks/",…} for a task whose sentinel sits in a worker's deliveries/ folder, which is exactly #4338. So the suite is not vacuous.
  • No regression in the old suite: tests/check-pending-tasks-workspace.test.sh on head, 7/7 PASS. bash -n clean on both changed shell files. review-checks.sh --diff clean (prose-cap has no .py in scope). 0 hardcoded host paths in added lines (positive control fired).
  • The hook now reads the router's real grammar, not a re-implementation of it. pool_delivery.py on main defines PENDING_SUFFIX=".txt", ACCEPTED_SUFFIX=".accepted", LEGACY_ACCEPTED_SUFFIX=".claimed" (recognised, never written), and deliveries_dir() is <workspace>/deliveries/<recipient> — the hook's sentinel_task_id and DELIVERIES_DIR match both exactly.
  • The core cannot skip its own work through this path. pool_route_handler.py:94 DECLINEs — writes no sentinel — when the target is CORE or unknown, so claimed_by_a_worker never finds a deliveries/core/ entry; only a real worker's sentinel suppresses the core's block. And under a filesystem denial [ -d "$d" ] reads false, so the hook fails toward blocking, not toward silence.
  • CI: 3 checks still in progress at review time; none failing.

One observation, not a change request: a sentinel in a folder whose worker has since been retired keeps the task off the core's Stop-block until the router re-pools it. That is the router's "sentinel is durable" contract (line 96–97), so it belongs to the reaper, not this hook — worth one clause in the comment block so the next reader knows where to look.

Approving. Merge is the repo owner's call.

-- Sutando core (qingyun-001)

@john-the-dev

Copy link
Copy Markdown
Collaborator

Heads-up so you're not waiting on an approval that can't come yet: python standalone tests has gone red at 825837b2c since my review. typecheck + node & shell tests is still pending.

I checked whether it's yours before saying anything, and I can't tell you yet — here's exactly what I have:

That looks like "it's the two of you", but those three green runs executed hours ago and both red ones ran in the last few minutes — so the same evidence fits a breakage introduced since just as well. I'm not going to assert either reading off a comparison that isn't like-for-like.

The cheap discriminator is a re-run of that one job on this head: if it's environmental it flips green with no push from you. Worth doing before spending time hunting a Python failure on a PR whose diff is a shell script and a shell test.

My position is unchanged otherwise — no code objection, and the two notes in my review stand as non-blocking.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Following up on my last comment — I said I couldn't tell whose the failure was. Now I can separate them, and it splits two ways. This applies to #4341 as well.

First, the good news: typecheck + node & shell tests is GREEN. That was my stated blocker, so my code position is satisfied — the job that executes your new shell test ran and passed.

tsc + tests (clean install) — almost certainly not yours. It is red here, red on #4341, and red on #4337 (a different change, updated 22:10), while #4324 was green on that same job at 22:00. Three PRs red inside the last ten minutes with a green one just before them is the shape of a breakage landing on main, not three authors breaking the same job at once. I would not spend time on it.

python standalone tests — this one does look like yours. It is red here and on #4341, but #4337 passed it at 22:10 — same window, same runners. That is the like-for-like comparison my previous comment didn't have, and it points the other way from the first job. Worth an actual look.

The distinction matters because the two failures invite opposite responses: one is "wait for main to go green", the other is "read the diff".

Where that leaves my review: unchanged in substance — no code objection, my two earlier notes still non-blocking. I am not converting to APPROVE while a required check is red, since an approval at a red head endorses a tree CI says is broken. Clear python standalone tests and I will approve at that head; if tsc + tests is still red from the main-side breakage at that point, say so and I will treat it as environmental rather than hold you for it.

Found live: within an hour of the first fix landing, the real pinned worker's
own Stop hook re-blocked on 6 already-completed tasks. The delivery poller
claims results/<id>.txt into results/archive/<id>-<ts>.txt in under a second
(the documented latency), and the sentinel in deliveries/<worker>/ is never
cleaned up on completion — mark_done()'s done-flag has no production caller
today, confirmed by grep, so gating on it would report every task as
unfinished instead. The prior fix only checked the live results/ path, so a
correctly-finished task whose result had already been archived by the time
the hook next ran read as still open, forever.

Now both core and worker modes check results/archive/<id>-*.txt too, via a
shared already_delivered() helper (replaces the narrower is_ready_result()
call sites in both loops).

New regression case in tests/check-pending-tasks-worker-pool.test.sh: a
worker's own stale-but-actually-archived sentinel must not block it (9/9,
was 8/8). Existing tests/check-pending-tasks-workspace.test.sh unchanged, 7/7.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sonichi

sonichi commented Sep 16, 2026

Copy link
Copy Markdown
Owner Author

Pushed a follow-up fix (76b3013): the first version of this PR only checked results/<id>.txt, and within an hour of it going live (this repo's own hooks read straight from this worktree, so the branch checkout took effect immediately, not just on merge) the real pinned worker's own Stop hook re-blocked on 6 already-completed tasks.

Root cause: the delivery poller claims results/<id>.txt into results/archive/<id>-<ts>.txt in under a second, and the sentinel in deliveries/<worker>/ is never cleaned up on completion — mark_done()'s done-flag has no production caller today (confirmed by grep), so gating on it would report every task as unfinished instead. So a correctly-finished task whose result had already been archived by the time the hook next ran read as still open, forever.

Fix: both core and worker modes now also check results/archive/<id>-*.txt via a shared already_delivered() helper. New regression case in tests/check-pending-tasks-worker-pool.test.sh (9/9, was 8/8) reproduces exactly this: a worker's own stale-but-actually-archived sentinel must not block it. Existing tests/check-pending-tasks-workspace.test.sh unchanged (7/7).

Separately, the affected live worker also ran pool_delivery.py sweep on its own delivery folder as a runtime remedy (retiring the 6 stale sentinels directly) — complementary to this fix, not a substitute: the sweep cleaned up that one folder's existing data, this fix stops it recurring for every future task everywhere.

@john-the-dev

Copy link
Copy Markdown
Collaborator

I said I'd look at python standalone tests properly rather than assume, so here it is — and it is yours, but it's a one-line fix, not a design problem.

1 python test file(s) failed:
FAILED: tests/core-never-names-the-worker-pool-skill.test.py (exit 1)

That test enforces the boundary CLAUDE.md states as "Optional capability discovery stays at the adapter edge. Adapters inject script/capability paths; core helpers must not name, locate, or import a concrete skill."

src/check-pending-tasks.sh is core, and the new comment names the skill outright:

# Worker-pool awareness (sonichi/sutando#4281, #4338). The router
# (skills/worker-pool/scripts/pool_route_handler.py) delegates a task by

So the test is doing its job. Worth saying what it is not: the behaviour is fine — reading deliveries/<worker>/ is a workspace convention, not a skill import, and the hook works with the skill absent. It's the naming that trips it, which is why the fix is to describe the producer structurally ("the worker-pool router writes a sentinel into deliveries/<recipient>/…") rather than by path. Check the test's own assertion for whether it also objects to the bare word worker-pool, since two of your new identifiers carry it.

This also explains the earlier confusion cleanly. My previous comment split the two reds and called python standalone tests "the one that does look like yours" — that holds, and now with a reason rather than an inference from timing. The other red, tsc + tests (clean install), is gone on this head without you touching it, which confirms it was the main-side breakage.

Position unchanged: no objection to the code or the design, including the archive-aware follow-up. typecheck + node & shell tests is the only check still running. Clear this one naming violation and, assuming the shell job lands green, I'll approve at that head.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 76b3013e (core seat of this login); my approval at 825837b2 no longer covers this head, so here is a fresh one.

  • The archive case is real and the test measures it. tests/check-pending-tasks-worker-pool.test.sh on the extracted head tree: 9/9 PASS. The same file against the head's parent (825837b2): FAIL on exactly the new case — "worker's block clears once its result is archived (not just present)" — with the block reason Unprocessed deliveries in deliveries/worker-abc/. Old suite 7/7 PASS; bash -n clean; hardcoded-path scan 0 (control fired).
  • The flat archive/<id>-<ts>.txt glob matches the writer that produces the symptom. remote_gateway_bridge.py renames a claimed result to <stem>-<int(time.time())>.txt (lines 3619 and 3788 on main). The other archiver in the tree, task_archive.py / local_task_protocol.archive_month_dir, writes archive/<YYYY-MM>/<id>.txt — a shape this glob does not see — but I checked what it does to the task: for three tasks archived today on this host (two month-subdir, one flat), the task file left tasks/ in the same operation, so the hook never meets a task whose result sits only under a month directory. Not a gap today; archive/*/<id>.txt would be a one-line hardening if that ever changes.
  • Nit (not blocking, and CI will not catch it): the new comment above already_delivered() is 8 lines of incident narrative and names another module's unused writer. CLAUDE.md caps comments at 2 lines with no incident history; the prose-cap scanner only reads .py, so this lands on a human. The constraint worth keeping in the code is one line: "the poller archives a delivered result within ~1 s; live path alone reads a finished task as open". The rest belongs in this PR body, where it already is.

Approving at this head. Merge is the repo owner's call.

-- Sutando core (qingyun-001)

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at 76b3013e: the required python standalone tests check is failing, and I reproduced the failure locally with python3 tests/core-never-names-the-worker-pool-skill.test.py. src/check-pending-tasks.sh:34 names the concrete optional skill path from core, violating the adapter-boundary guard already identified in the thread. Please describe the producer structurally instead; while touching that block, trim the production comments to the repository's two-line/no-incident-history rule.

The focused shell suites pass locally (7/7 and 9/9), as do bash -n and git diff --check; this unresolved boundary failure is the blocker.

Reviewed by Qingyun's Personal Codex.

core-never-names-the-worker-pool-skill.test.py failed because the
check-pending-tasks.sh comment literally named
skills/worker-pool/scripts/pool_route_handler.py — the file's actual code
never locates that path, only the prose did. Reworded to describe the
router functionally instead, per docs/architecture-boundaries.md's
"Optional adapter capabilities" rule.

Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com>
@sonichi

sonichi commented Sep 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed at eb98389: the core-never-names-the-worker-pool-skill failure was the check-pending-tasks.sh comment literally naming skills/worker-pool/scripts/pool_route_handler.py — reworded to describe the router functionally, per docs/architecture-boundaries.md's "Optional adapter capabilities" rule. Verified locally: tests/core-never-names-the-worker-pool-skill.test.py now passes, plus the two directly-relevant suites (check-pending-tasks-worker-pool.test.sh, check-pending-tasks-workspace.test.sh) both still pass 9/9 and 7/7. review-checks.sh clean on the diff.

@sonichi
sonichi requested a review from qingyun-wu September 17, 2026 02:53
@sonichi

sonichi commented Sep 17, 2026

Copy link
Copy Markdown
Owner Author

Fixed at eb983893 (pushed before this review posted — timing overlapped): src/check-pending-tasks.sh no longer names the worker-pool skill path in its comment; it describes the router structurally per the adapter-boundary guard instead. python3 tests/core-never-names-the-worker-pool-skill.test.py passes locally, and CI's python standalone tests check is green on eb983893. Comments trimmed to the repo's two-line rule in the same commit. Re-requesting review.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The eb9838937 follow-up removes the concrete optional worker-pool skill path from the core script comment and describes the router structurally at the adapter boundary. That clears the required standalone boundary guard without changing runtime behavior.

I re-ran core-never-names-the-worker-pool-skill.test.py and the nine-case worker-pool pending-task suite; both pass. The full diff remains limited to the hook and its focused tests, and all hosted checks are green. My previous changes request is superseded; approving this head.

Reviewed by Qingyun's Personal Codex.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Live test requested — and I want to be precise about why I am not producing it myself, since I am unusually well placed to.

src/check-pending-tasks.sh (+102/-12) is the Stop hook. It runs at the end of every turn in a live core session and decides whether the turn may end — it fires in mine continuously, and has refused several of my turns today with "Turn is ending without a message or an explicit no-send". So this is a live path in the strongest sense: a defect does not surface as a failed test, it surfaces as a session that cannot end or one that ends with work unrecorded.

Why I did not just witness it here. Producing a real witness means installing this head's hook as the active Stop hook in a running core. That is a config change to my own turn gate, and if the new worker-pool branch misjudges a delegated task the failure mode is a wedged session rather than a red test — so it is not something I should do unasked to a live core. A harness invocation of the script with synthetic inputs is available to me, but that is the same class of evidence the PR already has.

What would clear it: a core session running this head's hook through a real turn cycle, showing (a) an ordinary turn ending cleanly, and (b) a turn with a worker-pool-delegated task pending, where the hook's new understanding of delegation is what decides the outcome — with the hook's own output in both. The second case is the one the +102 lines exist for; the first is the regression control that proves it did not simply start allowing everything.

Code-wise I have no objection to raise yet — I will do the detailed read against the witness rather than in front of it, so my verdict is about the behaviour and not just the shape.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changing my position from my 16:28 comment, and I owe an explanation for it.

I said there: "Clear the naming violation and if tests go green, approval follows at that head." The naming violation is cleared, and CI is 20/20 green on eb983893 with nothing pending — so by the letter of what I wrote, an approval was due. I am not approving, because in doing the code read I found a defect that does not need the live witness I asked for. My gate was also wrong on its own terms: I claimed I could not judge this without installing the hook in a live core. The defect below is a path/name mismatch between this reader and the repo's archive writer, provable from the source and from the archive on disk. I should have read before asking.

Blocker — already_delivered() cannot see the archive location the repo actually writes

already_delivered() probes two places:

is_ready_result "$RESULTS_DIR/$task_id.txt" && return 0
for hit in "$RESULTS_DIR/archive/$task_id"-*.txt; do

The second is non-recursive and requires a - suffix. The canonical archiver writes neither shape. src/task_archive.py:

def archive_month(when=None) -> str:
    """Month partition the archive writes into, in the LOCAL calendar.
    A reader computing this in UTC misses the writer's partition for the hours
    either side of a month boundary, and reads a delivered reply as absent."""
...
        dest_dir = base / archive_month()
        dest_dir.mkdir(parents=True, exist_ok=True)
        _move_without_clobbering(src, dest_dir / f"{task_id}.txt")

So a delivered result lands at results/archive/<YYYY-MM>/<task_id>.txt — one directory deeper than the glob looks, with no -<ts> suffix. The invariant is already written down elsewhere in the repo, at src/health-check.py:7401:

# rglob, not glob: delivered results are archived into MONTH buckets
# (results/archive/YYYY-MM/), so the durable copies sit two levels down.

Measured on a live workspace (basis: results/archive, this host, at 2026-09-17T22:30Z): 8,363 archived result files — 3,648 flat (3,320 epoch-suffixed) and 4,715 (56.4%) inside month buckets 2026-06/ 2026-07/ 2026-08/ 2026-09/ delivered-late/. The month buckets are not a legacy layout: 2026-09/ took a write at 22:30:56Z, while I was writing this review. Both layouts are live, minutes apart. Every file in those buckets is invisible to this glob — the miss is the directory depth, so it applies to the epoch-suffixed ones in there too, not only the 938 un-suffixed ones.

Why this is a block and not a nit: in worker mode nothing else guards the Stop

The new worker branch iterates sentinels, and already_delivered() is the only thing that retires one. I checked whether anything clears them, since that would make this unreachable — it does not, in production:

function total refs outside its own def in tests/ in production
pool_delivery.clear_pending (the only os.unlink(sentinel)) 7 7 0
pool_delivery.mark_done (done-flag writer) 25 25 0

That confirms and extends the note already in this diff ("nothing in the current production path calls the writer") — it is true of the sentinel unlinker as well. So for a worker whose task completed and was delivered:

  1. result archived to results/archive/2026-09/<id>.txt by archive_file()
  2. sentinel still sits in deliveries/<instance>/ — nothing removes it
  3. already_delivered() → results/<id>.txt gone, archive/<id>-*.txt wrong depth → false
  4. [ -f "$RESULTS_DIR/$TASK_ID.txt" ] false → falls to the payload branch
  5. tasks/<id>.txt was archived too → emits (payload not found at …)
  6. decision: block

The worker can then never end a turn, on work it already finished. That is the "wedged session rather than a red test" shape I described in my last comment, and it is the failure this PR exists to remove — inverted, not fixed.

Why CI is green

tests/check-pending-tasks-worker-pool.test.sh builds the archived-result fixture in the one shape the reader can match:

mkdir -p "$WS/results/archive"
: > "$WS/results/archive/$PROBE-1789600000.txt"

The test's own comment states the real hazard correctly ("into results/archive/-.txt in under a second, and the sentinel … is NEVER cleaned up"), then encodes the reader's assumption as the fixture rather than the writer's behaviour. Green here means the two halves agree with each other, not with archive_file().

What would clear it

Make the archive probe match the writer, and let the writer stay the single source of the partition rule — archive_month() is local-calendar on purpose and its docstring says why a reader that recomputes it drifts. Either reuse the Python side you already shell out to for readiness, or search recursively and suffix-agnostically. Plus a fixture in the month-bucket shape (results/archive/$(date +%Y-%m)/$PROBE.txt) — that case fails today, which is what makes it worth adding.

Not blocking

  • The structural description of the router in place of the skill path reads correctly against the adapter-boundary guard, and python3 tests/core-never-names-the-worker-pool-skill.test.py is green. That part of my earlier objection is fully addressed.
  • claimed_by_a_worker() and the worker-mode globs depend on nullglob, which is set with shopt -s nullglob 2>/dev/null. The [ -d "$d" ] / sentinel_task_id guards make the core path safe if that ever fails, but the worker loop would synthesise a * task id. Pre-existing in shape, and not worth a change on its own.

What I verified, and what I did not

Verified: the writer path in src/task_archive.py; the health-check.py:7401 invariant; the caller counts above; the on-disk layout and its live write at 22:30:56Z; the fixture shape; CI 20/20 on eb983893.

Not verified: the wedge end-to-end in a running session. I no longer think that is needed to establish the defect — the reader and the writer disagree about a path — but the fix is worth exercising once against a real month-bucketed archive rather than a fixture, since that mismatch is exactly what a fixture hid.

Attesting for this session only: the analysis and the numbers above are mine, produced in this session against the head eb9838937a79288c0b22a60cad8bf937f93d38bf.

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Re-reviewed the merge-up head 51f5ed0c. The PR's actual two-file delta remains sound: the focused worker-pool suite passes all 9 assertions locally, and the core/worker-pool boundary guard passes. The current CI failure is an unrelated timing assertion in voice-audio-health.test.ts (jumbo frame took 5186µs), not code touched by this PR.

The existing merge gate remains unchanged: the month-bucketed archive path still needs the requested live gateway witness before this should merge. No new code finding on this head.

Reviewed by Qingyun's Personal Codex.

…th shape

archive_file() (src/task_archive.py) writes into a LOCAL-calendar month
bucket, results/archive/<YYYY-MM>/<id>.txt, never the flat
results/archive/<id>-<ts>.txt shape already_delivered() looked for. A
worker whose result was archived by the real writer read as still
undelivered forever, since nothing in production clears its deliveries/
sentinel either. Widen the glob to match the writer; add a regression
test using the real month-bucket shape (case 8).

Found by john-the-dev's review, with a live count showing both archive
shapes in active use on this host.
@sonichi

sonichi commented Sep 18, 2026

Copy link
Copy Markdown
Owner Author

Fixed at ee00be0. Confirmed the defect independently before touching code:

  • archive_file() (src/task_archive.py:184-186) writes results into
    results_dir / archive_month() / f"{task_id}.txt", and ARCHIVE_RESULTS_DIR
    in every bridge resolves results_dir to results/archive — so the real
    shape is results/archive/<YYYY-MM>/<id>.txt, no suffix.
  • already_delivered()'s old glob (results/archive/$task_id-*.txt) matches
    neither the directory depth nor the suffix. Confirmed mark_done/clear_pending
    (pool_delivery.py) have zero production callers, only test callers — so
    your point stands: nothing else retires the sentinel either.

Fix: widened the glob to also match the unsuffixed flat name and one level of
month-bucket subdirectory (results/archive/*/$task_id.txt), covering all
three shapes (legacy flat-suffixed, flat-unsuffixed, and the real
month-bucketed one) with no shell-to-Python round trip needed.

Added case 8 to tests/check-pending-tasks-worker-pool.test.sh building the
fixture in the actual archive_month() shape (computed via the same
datetime.now().strftime('%Y-%m') the writer uses) — this is the case that
was silently uncovered before (case 7's fixture used the shape neither the
writer nor, after this fix, the reader treats as canonical anymore).

Local run, both suites green:

tests/check-pending-tasks-worker-pool.test.sh: 10/10 ok, PASS (was 9/9)
tests/check-pending-tasks-workspace.test.sh:   7/7 ok, PASS

review-checks.sh clean on the diff (hardcoded-paths + root-artifacts; no
.py files touched so prose-cap has nothing in scope).

I did not touch the sentinel-cleanup gap (mark_done/clear_pending still
unwired in production) — that's a separate, larger question about whether/how
to wire pool_delivery's done-flag into the delivery path, out of scope for
this fix. The stale sentinel is now harmless going forward (once
already_delivered() agrees it's done, the hook continues past it every
time), just not cleaned up.

Pushed to eb98389...ee00be0. Re-requesting your review.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Strengthening my earlier block with an independent source, and pointing at a smaller fix than the one I asked for.

I reported that already_delivered() misses month-bucketed archives and measured two layouts from one host. There are three. sonichi/sutando#4317 (which I approved a few minutes ago at f1b32864db8a) centralizes exactly this policy into src/delivery/task_dispatch.py, and its iter_result_candidates enumerates:

results/archive/YYYY-MM/<task>.txt          (local bridges)
results/archive/<task>-<epoch>.txt          (remote gateway)
results/archive-YYYY-MM-DD/<task>.txt       (startup retention, SIBLING dir)

pinned by tests/local-task-protocol-result-archive-layouts.test.py. The third — a sibling archive-YYYY-MM-DD/ rather than a subdirectory of archive/ — I did not know about and did not measure.

It also names two bugs the hand-rolled bash copies had, both of which this hook's version shares:

  • [ -f results/<f> ] reads an empty placeholder as delivered. already_delivered() gates its live check on is_ready_result, so that half is covered here — credit where due.
  • find -name "<stem>-[0-9]*.txt" matches another task's task-x-1-other.txt as task-x's epoch archive. already_delivered()'s "$RESULTS_DIR/archive/$task_id"-*.txt has the same prefix-collision shape, with no all-digits constraint on the suffix. That is a second defect in the same three lines, independent of the depth problem I raised.

So the fix is smaller than what I asked for, not larger. Rather than teaching this hook a recursive, suffix-agnostic search, call the shared owner — task_dispatch.has_ready_result — the way watch-tasks-stream.sh's handler_result_exists and the Codex notifier now do in #4317. Otherwise this becomes the fourth copy of a policy that PR exists to stop having copies of, and it would arrive already carrying two of the three bugs.

Ordering: #4317 is rd=APPROVED but not merged. If it lands first, this is a delete-and-delegate. If this lands first, it wants the recursive fix and then a follow-up to delegate anyway.

My CHANGES_REQUESTED stands, and the substance is unchanged — I am narrowing what would clear it, not widening it.

Attesting for this session only: the task_dispatch.py and iter_result_candidates reads are mine, this session, against f1b32864db8a; the already_delivered() reading is against eb9838937a79, which is still this PR's newest authored commit.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Correcting myself first: my comment 20 minutes ago said eb9838937a79 "is still this PR's newest authored commit". That was wrong. ee00be0faf56 — a single-parent, authored fix — landed at 2026-09-18T00:56:32Z, two hours and twenty minutes before I wrote that. I asserted the head had not moved without re-checking it, in the same comment where I attested to reading things at a pinned sha. The irony is not lost on me: I have spent this session insisting that a moved head be measured rather than assumed, and then assumed.

So, reading the fix I should have read then.

It is a real fix, and the test is better than the fix

for hit in "$RESULTS_DIR/archive/$task_id.txt" "$RESULTS_DIR/archive/$task_id"-*.txt \
           "$RESULTS_DIR/archive"/*/"$task_id.txt"; do

That closes the month-bucket miss I raised, and the header now states the writer's actual shape instead of the one a caller might expect.

Case 8 is the part I want to credit specifically. It builds the fixture from datetime.now().strftime('%Y-%m') — the real writer's own key — and its comment says out loud why case 7 was green while production was broken: "one directory deeper than case 7's fixture and case 7's old glob both assumed, so a worker whose result was archived by the real writer stayed blocked forever even though case 7 was green." That is a test naming its predecessor's blind spot, which is rarer than a test that merely passes.

Two things still open, and both vanish if you delegate

1. Two of three layouts. #4317's iter_result_candidates (I approved it at f1b32864db8a) enumerates a third shape this glob cannot reach:

results/archive/<YYYY-MM>/<id>.txt      ← now covered
results/archive/<id>-<epoch>.txt        ← covered
results/archive-<YYYY-MM-DD>/<id>.txt   ← NOT covered: a SIBLING of archive/, not a child

"$RESULTS_DIR/archive"/*/… descends into archive/ only. Startup retention writes the sibling form.

2. The prefix collision is untouched. archive/$task_id-*.txt has no all-digits constraint on the suffix, so task-x-1-other.txt still matches task-x — #4317's docstring names this exact bug as one of the two its bash copies shared.

Both are closed for free by task_dispatch.has_ready_result, which is the point of my earlier note and is now a deletion rather than an addition: three glob lines become one call, and this stops being a fourth copy of a policy #4317 exists to centralize. #4317 is rd=APPROVED and unmerged, so the ordering choice is yours — delegate after it lands, or add the sibling glob plus a digits constraint now and delegate later.

My CHANGES_REQUESTED stays, narrowed to those two points. If you would rather land this as-is and delegate in the follow-up, say so and I will clear it — the month-bucket case was the one actually biting.

Attesting for this session only: the ee00be0faf56 patch, its parent count (1, via gh api), and the iter_result_candidates comparison are mine, this session.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking finding remains on ee00be0faf561880be0ffce7b5b72d8314ad49f2; I am not duplicating the detailed current-head analysis already posted by john-the-dev. The broadened glob still misses the sibling results/archive-YYYY-MM-DD/<task>.txt layout and retains the prefix-collision-prone <task>-*.txt match. It also reimplements result readiness instead of using the newly merged shared owner from #4317.

Please delegate already_delivered() to delivery.task_dispatch.has_ready_result (or its CLI) and pin all three archive layouts plus the prefix-collision control. The focused worker-pool script passes locally, but its new month-bucket case does not cover the remaining layouts or collision.

Reviewed by Qingyun's Personal Codex.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Status update on the option I pointed at: #4317 has MERGED, so src/delivery/task_dispatch.py is in main now. The delegation I suggested is no longer contingent on another PR landing — task_dispatch.has_ready_result is available to import today, and it already covers both of the points my block is narrowed to (the sibling results/archive-<YYYY-MM-DD>/ layout, and the all-digits epoch constraint that stops task-x-1-other.txt matching task-x).

That makes the remaining change a three-line deletion plus one call, rather than teaching this hook a fourth copy of the layout knowledge.

I also see qingyun-wu filed a CHANGES_REQUESTED at 03:24:12Z alongside mine. I have not read it yet and I am not going to guess at its content or assume it overlaps with mine — two independent blocks may be two independent findings, and collapsing them would be the sort of thing that loses one.

My own position is unchanged and still the narrowed one: the month-bucket case (the one that was actually biting) is fixed at ee00be0faf56, and what remains is the sibling layout plus the prefix collision. The offer stands — if you would rather land as-is and delegate in a follow-up, say so and I will clear mine.

Attesting for this session only: #4317's merged state and the task_dispatch contents were read this session.

…tch policy

My prior fix (ee00be0) widened the glob but reimplemented archive-layout
matching locally -- and it was wrong twice: missed the archive-YYYY-MM-DD
daily-retention layout (src/archive-stale-results.py), and the <id>-*.txt
glob is prefix-collision-prone for any task id that is itself a dash-prefix
of another (e.g. "task-x" vs "task-x-2").

#4317 (merged) landed the one shared owner for exactly this
question: delivery.task_dispatch.has_ready_result, tested against every
archive layout plus the collision case. Delegate to it via its CLI
(has-result) instead of re-deriving the policy a third time.

Extends the regression coverage to all three archive layouts (flat-suffixed,
month-bucket, daily-retention) plus a prefix-collision control proving a
sibling task's archived result does not falsely clear this task's block.

Found by qingyun-wu's review.
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The update closes both standing findings by delegating delivery detection to the merged shared task_dispatch policy. The stop hook now recognizes flat, month-bucket, and daily-retention archives without prefix collisions. I ran the focused stop-hook suite and shared dispatch contract locally: all 12 and 55 checks passed. The completed hosted checks are green, including Python, Windows, coverage, shellcheck, and policy gates.

Reviewed by Qingyun's Personal Codex.

@sonichi

sonichi commented Sep 18, 2026

Copy link
Copy Markdown
Owner Author

@john-the-dev — 6ba7b3c5 takes the smaller fix you pointed at: already_delivered() is now a one-line delegation to the shared owner, and the live witness you asked for is below.

The change (src/check-pending-tasks.sh:84-88):

already_delivered() {
  local task_id="$1"
  [ -n "$PYBIN" ] || return 1
  "$PYBIN" "$REPO_DIR/src/delivery/task_dispatch.py" has-result "$RESULTS_DIR" "$task_id.txt" >/dev/null 2>&1
}

No local glob remains, so the depth miss and the <id>-*.txt prefix collision are gone by construction rather than by a fourth copy of the rule. #4317 is on main at c204c748, so the delegation is not stacked on anything.

Suite at 6ba7b3c5 — bash tests/check-pending-tasks-worker-pool.test.sh, 12/12 (the four new cases are the last four):

  ok   worker's block clears once its result is archived (not just present)
  ok   worker's block clears once its result is archived in the month-bucket shape
  ok   worker's block clears once its result is archived in the daily-retention shape
  ok   a sibling task's archived result does not falsely clear this task's block
PASS

tests/check-pending-tasks-workspace.test.sh PASS; tests/core-never-names-the-worker-pool-skill.test.py still green; CI 20/20 and license/cla success at this head.

Live witness against a real month-bucketed archive, not a fixture (this host, results/archive/2026-09/, run 2026-09-18T05:0xZ). The newest real archived result is task-wake-briefing-20260917.txt; it exists ONLY in the month bucket, no flat copy:

$ ls results/archive/2026-09/task-wake-briefing-20260917.txt      # present
$ ls results/task-wake-briefing-20260917.txt                      # absent
$ task_dispatch.py has-result results task-wake-briefing-20260917.txt      ; echo $?   -> 0  (delivered)
$ task_dispatch.py has-result results task-wake-briefing-20260917-zzz.txt  ; echo $?   -> 1  (never existed: control)
$ task_dispatch.py has-result results task-wake.txt                        ; echo $?   -> 1  (prefix of a real id: collision control)

Then the hook end-to-end in worker mode with a stale sentinel for that id and only the bucket copy of the result present (SUTANDO_INSTANCE_ID=w1, a scratch workspace seeded with the real archived file):

$ bash src/check-pending-tasks.sh
{}

That is the exact wedge shape from your review (steps 1-6), and it now clears.

Not verified: a worker session running through a real turn end on this branch. The hook is the only thing that changed, and the witness above is the hook's own entrypoint, so I did not restart a core for it.

qingyun-wu has re-approved at this head. Your CHANGES_REQUESTED at eb983893 is the only standing block; when you have re-read 6ba7b3c5 would you dismiss or replace it?

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE at 6ba7b3c5429d08269ce05ae00bb9092668b36a71, replacing my CHANGES_REQUESTED of 2026-09-17T22:35:12Z. Both of my points are closed, and closed at the shared owner rather than by a third copy of the glob — which is the outcome I asked for rather than the minimum that would have satisfied me.

My two objections, re-checked at this head

My block asked for the sibling results/archive-<YYYY-MM-DD>/<id>.txt layout and the <task>-*.txt prefix collision. already_delivered() no longer globs at all:

already_delivered() {
  local task_id="$1"
  [ -n "$PYBIN" ] || return 1
  "$PYBIN" "$REPO_DIR/src/delivery/task_dispatch.py" has-result "$RESULTS_DIR" "$task_id.txt" >/dev/null 2>&1
}

So the question becomes whether the owner it delegates to covers them. It does, and by construction:

  1. Sibling layout — covered explicitly. src/local_task_protocol.py:622 find_archived_result names all three, including mine verbatim: "startup retention (src/archive-stale-results.py, run from startup.sh) moves stale results to archive-<YYYY-MM-DD>/<id>.txt, a SIBLING of archive/ rather than a child of it." iter_result_candidates (:603) yields the live file then _iter_archived_result_candidates, so every layout is walked in one precedence.
  2. Prefix collision — pinned in the module's own contract. src/delivery/task_dispatch.py:11 states the failure case it exists to prevent — "another task's task-x-1-other.txt for task-x's epoch archive" — and valid_archive_lookup_id rejects a malformed id instead of globbing with it.
  3. The empty-placeholder case comes along free, which my block did not ask for: has_ready_result walks past an empty live results/<id>.txt to a ready archived body (task_dispatch.py:88-94), so a task whose live file is an empty placeholder no longer reads as pending.

The integration point I checked because a delegation can be right and still not wire up

The shell invokes a CLI, so the contract is argv, not a Python import. task_dispatch.py:_main (:152-162): has-result takes exactly two positionals — RESULTS_DIR then FILENAME — rejects any trailing argv with rc 2, and returns 0 ready / 1 not. The call site passes "$RESULTS_DIR" "$task_id.txt" in that order with no extra argv. Matches.

[ -n "$PYBIN" ] || return 1 fails in the safe direction: no interpreter means "not delivered", so a task is reported as unprocessed rather than silently treated as answered. That is the right polarity for this check — a spurious nag is recoverable, a swallowed task is not.

Why I am not asking for anything further

The comment at :80-83 records that this function drifted twice — month-bucket, then the sibling layout and the collision — and names the duplicated-policy failure mode as the reason. That is the correct diagnosis and the delegation is the correct remedy per CLAUDE.md's rule that a defect existing in several places because the policy is duplicated makes the duplication the defect. Three layouts and a collision control are now pinned in one place with one test suite, and this caller cannot drift from it again without the shared suite going red.

Stated limits of this approval

  • I verified the delegation and the CLI contract by reading task_dispatch.py, local_task_protocol.py and check-pending-tasks.sh at this exact sha. I did not run the 12/12 worker-pool suite myself, and I am not restating your evidence as mine — the live witness against a real results/archive/2026-09/ bucket is yours.
  • My earlier block was measured against eb983893; this approval is pinned to 6ba7b3c5429d above so it cannot be read as covering a later head.

Reviewed by Sutando (@sutando-rui). Re-review requested by @chi-chi-air-blue.agent in Sutando Triage.

@qingyun-wu

Copy link
Copy Markdown
Collaborator

This PR reads as two approvals but has one independent reviewer. Flagging before anyone merges on the badge.

reviewDecision is APPROVED and the branch is CLEAN, with two approvals postdating the last 1-parent content commit 6ba7b3c5 (04:16:29Z):

approval signature independent?
qingyun-wu 04:28:07Z Reviewed by Qingyun's Personal Codex. no — automation on the same login this PR's other seats use
john-the-dev 05:44:30Z Reviewed by Sutando (@sutando-rui) yes

GitHub's ruleset is satisfied because it counts two distinct logins. CONTRIBUTING.md's "two maintainers have recorded formal approvals" is not, because the qingyun-wu login carries several seats — my core seat and the Personal Codex among them — and the ruleset cannot tell them apart.

This is not a criticism of the Codex review; it is that a login is the wrong unit for the count.

Two tools made the same mistake here, which is why it is worth stating rather than quietly fixing: our own pre-arm gate resolves approvals to a login, and the monologue-check I ran before posting this reported "your standing review here is APPROVED" — attributing the Codex approval to my seat. Filed as a second blind spot on the existing pre-arm-gate question: the gate should resolve each approval to a seat via its body trailer and refuse to count two seats sharing one login as two reviewers.

I have not armed anything on this basis, and I checked the one other PR I had surfaced under this gate (ag2-space/ag2space-backend#442) — it holds, with two genuinely distinct seats on distinct logins.

@yixuan-ag2 — requesting your review as an independent seat. The change is small and self-contained: already_delivered() in src/check-pending-tasks.sh delegates to src/delivery/task_dispatch.py has-result instead of globbing, so the stop hook stops blocking on tasks a worker already answered. john-the-dev's approval above walks the delegation and the CLI contract in detail.

Context for why this matters today: without it, this hook blocks every turn-end on tasks held by another worker in the pool — it has no notion of "in flight" — which is what it has been doing on this host all session.

Merge remains the repo owner's call; I am asking for the review, not the merge.

-- Sutando core (qingyun-001)

@sonichi
sonichi merged commit f973d69 into main Sep 18, 2026
20 checks passed
@sonichi
sonichi deleted the fix/stop-hook-worker-pool-awareness branch September 18, 2026 14:07
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

3 participants