Skip to content

fix(test): handler-terminal-rc waits on what the watcher writes, not on a window - #5007

Merged
sonichi merged 7 commits into
mainfrom
fix/flaky-watcher-handler-terminal-rc
Oct 7, 2026
Merged

sonichi merged 7 commits into
mainfrom
fix/flaky-watcher-handler-terminal-rc

Conversation

@sonichi

@sonichi sonichi commented Oct 1, 2026 •

Copy link
Copy Markdown
Owner

What changed, and why

tests/watch-tasks-stream-handler-terminal-rc.test.py failed on #4994 (run 36873678067, job 110408347985, python standalone tests (3/5), 2026-10-01 ~14:08Z) — a PR that touches no watcher file — and passed on rerun. The failing check:

→ tests/watch-tasks-stream-handler-terminal-rc.test.py (79s)
  FAIL control: an ordinary failure still falls back to the core — the fallback path was removed, not narrowed
  watcher stderr:
    watch-tasks-stream: role=standby inbox=/tmp/term-rc-9w2i4jna/ws/tasks pid=29890
    watch-tasks-stream: task handler in flight for task-demo.txt at shutdown; leaving /tmp/term-rc-9w2i4jna/ws/tasks/task-demo.txt to the next watcher's sweep
    watch-tasks-stream: task handler in flight for task-demo.txt at shutdown; leaving /tmp/term-rc-9w2i4jna/ws/tasks/task-demo.txt to the next watcher's sweep

The test polled a fixed 25s window per scenario, and the suite ran 79s where it normally runs ~55s: the control scenario (handler probes 0, exits 1, so the task must be handed to the core) used its whole window. What the stderr says: the claim was still held at the kill (the "in flight" line) and no handler verdict line was ever printed (optional task handler failed ... falling back to live core for rc 1, or handler timed out / required Team handler failed for the other branches). So the watcher got past acquire_task_claim and never reached the verdict inside run_handler_now — the synchronous handler run and its watchdog subshell — for at least 24s. The test could not say more: its loop could not tell a stalled watcher from one that had exited, two of the four scenarios had no stdout line to wait for and always ran the full 25s, and a FAIL printed only stderr.

This PR changes only the test. Every scenario now ends on something the watcher itself writes, with a bound a stall reaches rather than a window the normal case waits out:

  • rc 4 (must-handle): the published result file.
  • rc 1 (control): the TASK_FILE line on stdout.
  • rc 0 (control 2): the after-done transition (through the documented SUTANDO_WATCHER_TRANSITION_HOOK seam, the same one watch-tasks-stream-shutdown-leaves-inflight-claim.test.py uses) plus the claim's release.
  • restart witness: the first watcher's sentinel naming its pid, instead of time.sleep(2.0).
  • Any scenario also ends the moment the claim is released after after-handler — the watcher is done with the task whatever it decided — so a wrong outcome is diagnosed immediately, not at the bound.
  • A watcher that exits ends its wait at once (p.poll()), and a FAIL prints the transitions reached, the exit status, the claim and result state, stdout and stderr.

Watcher code is untouched. The root cause of the stall itself is not established: it did not reproduce locally (macOS, bash 5.3; see below), so this PR makes the test end on observables and leaves a diagnosable trace for the next occurrence rather than widening a window that a stall would not respect anyway. Suite cost drops from 55s to ~7s quiet (17–44s on a host at load average 30–50), which is the lever #4862 names for this family.

Update after review (cecaced00, e36c5d2d4, de2a75cce)

What the test now does on top of the description above (test only; git diff origin/main -- src/ is empty):

  • Reads on after settling. The rc 4, rc 0 and restart absence checks keep reading the watcher's stdout for 2.5 s after the task settles. The bound, as @john-the-dev stated it: an emit that lands later than 2.5 s after the claim is released still passes. No current watcher path emits asynchronously; the tail exists to catch a regression that does.
  • Asserts each settle point. The rc 4, rc 0 and restart cases each assert that their handler ran and their claim was released. The rc 0 control also asserts that it reached after-done.
  • Ends the restart wait early, the same way the other cases do.
  • Starts all scenarios together, so the tails overlap instead of adding up. The suite runs in about 12–13 s on a loaded host.
  • Cleans up after itself. The temp workspaces are removed at exit, whether the test passes or fails.

The mutation tables are in the PR comments: 5/5 of rui's mutations and both of @keweichen's controls fail at the new head.

The shard 3 failure at cecaced00 is a watcher finding, not a harness race. Filed as #5012. That job (110485063284) failed on the new rc 0 success control:

  FAIL control: a successful run records after-done and releases its claim — the success path did not complete, so 'emits nothing' would be vacuous
  waits ended by: ['exited after 0.0s', 'exited after 0.0s']; watcher exit status: 0
  transitions: ['1790874406 before-claim task-demo.txt']
  claims held: ['task-demo.txt']
  results: []
  stdout: b''
  watcher stderr:
    watch-tasks-stream: role=standby inbox=/tmp/term-rc-8d2amvcd/ws/tasks pid=23517
    watch-tasks-stream: task handler in flight for task-demo.txt at shutdown; leaving /tmp/term-rc-8d2amvcd/ws/tasks/task-demo.txt to the next watcher's sweep
    watch-tasks-stream: task handler in flight for task-demo.txt at shutdown; leaving /tmp/term-rc-8d2amvcd/ws/tasks/task-demo.txt to the next watcher's sweep

What this shows:

  • The watcher died on its own. It exited with status 0 before the test had signalled it: the first wait found it already gone.
  • It died mid-handler. Its last transition was before-claim, and it was still holding the claim.
  • The stderr matches the original refactor(skills): rename claude-gemini to agy; keep a one-release alias #4994 failure. In both, the "in flight" shutdown line is printed twice. That points at two processes running cleanup(). In a subshell, cleanup() uses the parent's $$, so it would kill -TERM 0 the watcher's whole group, and the watcher then exits 0 through its own trap. Which subshell runs the inherited trap, and why, is not established; it did not reproduce on macOS.

So the original flake this PR set out to fix is most likely #5012, a watcher bug. This PR does not fix it. What it changes is the test: that failure now shows up with its exit status, last transition and claim state, instead of as a 25 s timeout.

The other two failures at cecaced00 are unrelated to this file: python standalone tests (1/5) failed in tests/sparrow-result-guard.test.py; I did not read the tsc job's log. All checks on e36c5d2d4 passed.

Files to look at first

  • tests/watch-tasks-stream-handler-terminal-rc.test.py — Workspace (fixture + hook), Watcher.wait_until (observable / exited / timeout), run() (per-scenario settled predicate plus the claim-released end), restart_witness() (sentinel wait).

What this proves

The four assertions are unchanged (must-handle not emitted + failure published; rc 1 still falls back; rc 0 emits nothing; restart witness publishes, does not emit). One assertion is added to the restart witness — the first watcher stamped its sentinel before it was stopped — which the old sleep(2.0) only assumed.

Feature/documentation consistency

N/A — test-only change; no user-visible behaviour or document changes.

Before/after evidence

Parent commit (1be758ba5, = origin/main), the test as it was — passes locally but costs 55s because two scenarios wait out the window:

$ bash -c "env $U python3 tests/watch-tasks-stream-handler-terminal-rc.test.py"
  ok   a must-handle result is NOT emitted to the live core
  ok   a must-handle result publishes a terminal failure instead
  ok   control: an ordinary failure still falls back to the core
  ok   control: a successful run emits nothing and needs no failure
  ...
PASS — handler terminal rc outranks the probe-time disposition
bash -c "env $U python3 tests/watch-tasks-stream-handler-terminal-rc.test.py"  1.87s user 1.36s system 5% cpu 55.177 total

Reproduction attempt under load, parent commit (20 sequential runs, 10 × yes > /dev/null on a 10-core host, SUTANDO_ scrubbed via tests/fixtures/clean_watcher_env.py's set):*

run 1: PASS (55s) ... run 20: PASS (56s)
TALLY before: pass=20 fail=0 of 20 (load: 10 x yes)

The CI failure did not reproduce: 0/20 full-suite runs, and 0/80 runs of the control scenario alone (12s deadline) under the same load. A direct stress of the watchdog subshell pattern (( trap ... TERM; sleep & wait ) & then kill -TERM; wait) ran 1800 iterations under 60 busy loops with no lost signal and no slow wait on bash 5.3. The stall is Linux-only so far (ubuntu-latest, bash 5.2, four suites per 4-vCPU runner under coverage) and observed once.

HEAD (425c417e3), quiet host:

$ bash -c "env $U python3 tests/watch-tasks-stream-handler-terminal-rc.test.py"
  ok   a must-handle result is NOT emitted to the live core
  ok   a must-handle result publishes a terminal failure instead
  ok   control: an ordinary failure still falls back to the core
  ok   control: a successful run emits nothing and needs no failure
  ok   restart: the first watcher came up before it was stopped

  restart witness: watcher pid 86622 stopped, pid 86999 started; task arrived after the restart
    emitted to the live core: False
    published by the restarted watcher: ['task-restart.txt']
    result body: 'I could not safely process this Team-tier task because the restricted runtime failed. No unrestricted fallback was used.'
  ok   restart: the restarted watcher does NOT hand the task to the core
  ok   restart: the restarted watcher publishes a terminal failure
PASS — handler terminal rc outranks the probe-time disposition
bash -c "env $U python3 tests/watch-tasks-stream-handler-terminal-rc.test.py"  2.13s user 1.45s system 54% cpu 6.610 total

HEAD, 20 × under the same load (run twice; the second on the exact pushed commit):

TALLY after: pass=20 fail=0 of 20 (load: 10 x yes)     # 16–44s per run, host load average 30–50
TALLY after2: pass=20 fail=0 of 20 (load: 10 x yes)    # 11–19s per run

The diagnosis path, checked by mutation (not committed). Control handler changed to exit 4 — the FAIL now arrives when the claim is released (4.3s), not at the bound, and says where the watcher went:

  FAIL control: an ordinary failure still falls back to the core — the fallback path was removed, not narrowed
  waits ended by: ['seen after 4.3s', 'seen after 0.0s']; watcher exit status: 0
  transitions: ['1790868985 before-claim task-demo.txt', '1790868985 after-handler task-demo.txt']
  claims held: []
  results: ['task-demo.txt']
  stdout: b''
  watcher stderr:
    watch-tasks-stream: role=standby inbox=.../term-rc-hw72uu1s/ws/tasks pid=82339
    watch-tasks-stream: required Team handler failed for task-demo.txt (exit 4); publishing safe terminal failure

fswatch stub changed to exit 1 — the watcher's early exit ends the wait in 1.0s instead of 40s:

  FAIL restart: the first watcher came up before it was stopped
  waits ended by: ['exited after 1.0s']; watcher exit status: 1
  transitions: []
  ...
    watch-tasks-stream: fswatch failed to start; no sentinel written and no standby touched

Had the CI run carried this trace, the stderr above would have been accompanied by the last transition reached (before-claim with no after-handler = stalled in the handler run / watchdog wait) and the watcher's exit status.

Tests run

Adjacent watcher suites at HEAD, scrubbed env (tests/watch-tasks-stream*.test.{py,sh}, tests/watcher*.test.{py,sh}, 34 files): 33 PASS. tests/watch-tasks-stream-bare-directory-event.test.py FAILs one check on this host (a second EOF past the relaunch budget ends the watcher, status 1 — rc=None), in the chain run and standalone at load average 11; it is a file this PR does not touch, on a watcher this PR does not touch, and it is in #4862's list — reported here, not fixed here.

$ git diff origin/main...HEAD | bash scripts/review-checks.sh
prose-cap: PASS (comment blocks within cap 2, exts .py)
review-checks: PASS (hardcoded-paths + root-artifacts + prose-cap clean)

Live path: N/A (test only). Stacked PR: N/A. Hardcoded paths: none added (fixtures use tempfile.mkdtemp). tests/python-suite-costs.txt still lists this file at 25s (measured in CI); left for the next regeneration from a job log rather than hand-edited.

Refs #5012 (the watcher exiting mid-handler, seen in this test at cecaced00 and on #4994), #4862 (this file is one of the suites it lists; the others are not changed here), #4750, #4967. Not a duplicate of #5000, which isolates these suites on their own CI leg; this PR changes what the test waits on, which that lane change does not.

🤖 Generated with Claude Code

…on a window

The suite polled a fixed 25s window per scenario. Two of the four scenarios
(rc 4, rc 0) have no stdout line to wait for, so they always ran the full
window and the file cost 55s; the control scenario (rc 1) waited for the
TASK_FILE line but could not tell a watcher that stalled from one that had
exited, and a FAIL printed only the watcher's stderr. In CI (run 36873678067,
a PR touching no watcher file) the control scenario failed that way: the
claim still held at the kill, no handler verdict line, nothing to say where
the watcher stopped.

Every scenario now ends on something the watcher itself writes: the TASK_FILE
line, the published result, or the `after-done` transition plus the claim's
release (through the documented SUTANDO_WATCHER_TRANSITION_HOOK seam); the
restart witness waits for the first watcher's sentinel instead of sleeping
2s. A claim released after `after-handler` ends any scenario at once, so a
wrong outcome is diagnosed immediately rather than at the bound; a watcher
that exits ends its wait at once; and a FAIL prints the transitions reached,
the exit status, the claim and result state and the stderr. The bound (40s)
is a stall ceiling, not the expected cost.

Watcher code is untouched. Before: 20/20 PASS in 55s each under 10x CPU
load (the CI failure did not reproduce on macOS/bash 5.3). After: 20/20 PASS,
17-44s each under the same load on a host at load average 30-50, 7s quiet.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sonichi
sonichi requested a review from qingyun-wu as a code owner October 1, 2026 15:59
@sonichi
sonichi requested review from john-the-dev and jsun-m October 1, 2026 16:01
@github-actions

github-actions Bot commented Oct 1, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 91%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at exact head 425c417e3d539bf246a66fc8c748a2c4fbc9cff1, with one non-blocking gap that should be named.

Scope:

  • One file, tests/watch-tasks-stream-handler-terminal-rc.test.py (+197/−124). The watcher is unchanged.
  • No scenario dropped: rc=4, the rc=1 control, the rc=0 control and the restart witness all remain. The fixed sleep 2.0 before the restart is now an asserted "first watcher came up".
  • Five runs at head took 3–5 s, against 54 s for the old file. Every wait is bounded (40 s main, 10 s claim, 10 + 5 s stop).

Mutation results (watcher mutated, both test versions run, watcher restored):

mutation new test old test
rc 4 no longer outranks the stored disposition FAIL FAIL
every non-zero rc treated as must-handle FAIL (rc=1 control) FAIL
success path emits TASK_FILE right after releasing the claim FAIL (rc=0 control) FAIL
rc=4 path publishes a failure and emits FAIL FAIL
success path emits 2 s later, in the background PASS (survives) FAIL

The gap (nit, not blocking):

  • The absence checks — rc=4 "not emitted" (~241-243), rc=0 "emits nothing" (~256-258), restart "not handed to core" (~232-233) — now stop watching once the claim is released.
  • run_handler_now decides, emits or publishes, and then releases its claim (src/watch-tasks-stream.sh ~732-777). So a synchronous wrong emit is caught, and so is anything printed during shutdown, since stop() drains stdout.
  • An emit that happens after the release is not caught. The old 25 s window caught the delayed-emit mutation only incidentally, and no current watcher path emits asynchronously. Still, "nothing weakened" is slightly too strong.
  • A ~2 s read-only tail after settling, in those three cases, would kill that mutation and keep the suite fast.

Two more small ones:

  • run() discards wait_until's return (~201). For the rc=0 control, assert that after-done was reached. Otherwise a watcher that dies early makes "emits nothing" pass vacuously. The old file had the same gap.
  • The restart wait (~232) has no over() early exit, so a failing run spends the full 40 s there. Passing runs are unaffected.

Not verified: that this resolves the CI stall in #4862 (not reproducible locally), and Linux/CI behaviour under load.

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at exact head 425c417e3d539bf246a66fc8c748a2c4fbc9cff1.

Blocker — the refactor weakens the must-handle non-emission regression test.

run() now stops as soon as the result/claim-release observation completes (tests/watch-tasks-stream-handler-terminal-rc.test.py:189-205). Watcher.stop() then sends TERM to the whole watcher process group (:150-160), and only afterward does the assertion inspect captured stdout (:241-243). That can make the test suppress the very bad output it is meant to detect.

I verified this with an uncommitted production mutation placed immediately after the rc=4 claim release at src/watch-tasks-stream.sh:775:

if [ "$handler_rc" -eq 4 ]; then
  ( sleep 2; emit_fallback_task_file "$announce" ) &
fi

The watcher is otherwise byte-identical at the parent and HEAD. Results:

HEAD 425c417e3: PASS (6.6s)
  ok   a must-handle result is NOT emitted to the live core

parent 1be758ba5: FAIL
  FAIL a must-handle result is NOT emitted to the live core
    — the task reached the unrestricted core despite the handler refusing it

In a real continuously running watcher, that delayed TASK_FILE leaks the required-handler task to the unrestricted core. HEAD passes because the harness kills the delayed emitter immediately after release; the parent's observation window catches it. This is a direct mutation of the security-boundary behavior, so the PR's claim that the four assertions are unchanged is not preserved.

There is a second false-green in the same synchronization logic: both wait_until() results at lines 201 and 204 are discarded. An uncommitted mutation that exits successfully immediately after after-handler but before record_worker_done / after-done leaves the entire HEAD suite green, including the rc=0 control that says it reached after-done.

Required fix: preserve a short bounded read-only drain/quiescence tail for every negative-emission scenario (rc=4, rc=0, and restart), and require the expected settle observations instead of discarding their return values. Please rerun both mutations as controls. A small tail retains most of the measured runtime improvement (clean HEAD is about 6s versus about 55s at the parent).

Other evidence: the exact-head suite passes unmutated; git diff ... | bash scripts/review-checks.sh passes; current CI and SHA-bound CLA are green. No live-service restart is required for this test-only diff. Worst case as written is a false-green test while a must-handle task is handed to the unrestricted core.

Verdict: changes requested; the negative assertions must keep observing long enough to discriminate post-release output, and the expected settle event must be asserted.

Review follow-up, test only; the watcher is untouched.

- The absence checks (rc 4 not emitted, rc 0 emits nothing, restart not
  handed to the core) keep reading stdout for 2.5s after the claim is
  released, so an emit that lands after the release is caught again.
- The rc 0 control asserts `after-done` was reached and the claim released,
  so a watcher that dies early cannot pass "emits nothing" vacuously.
- The restart wait ends early once the claim is released after
  `after-handler`, like the other scenarios.
- The three scenarios and the restart witness's first watcher start
  together, so the tails overlap instead of adding to the suite time.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sonichi

sonichi commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

Follow-up to @john-the-dev's review, at cecaced00 (one commit, test only; src/watch-tasks-stream.sh unchanged — git diff origin/main -- src/ is empty).

What changed

  1. Late-emit gap. The three absence checks (rc 4 "not emitted", rc 0 "emits nothing", restart "not handed to the core") keep reading stdout for 2.5s after the claim is released. 2.5s rather than 2s so a 2s-delayed emit isn't a coin toss.
  2. rc 0 control can't pass vacuously. New check control: a successful run records after-done and releases its claim.
  3. Restart wait exits early once the claim is released after after-handler, like the other scenarios.
  4. To keep the tails from adding up, the three scenarios and the restart witness's first watcher now start at the same time (separate temp workspaces). Each tail is measured from the moment its own task settled.

Mutations (watcher mutated in a scratch worktree, test run, git checkout -- src/watch-tasks-stream.sh after each):

# watcher mutation result failing check(s)
1 rc 4 no longer outranks the stored disposition FAIL must-handle NOT emitted; publishes a terminal failure; restart does NOT hand to core; restart publishes
2 every non-zero rc treated as must-handle FAIL control: an ordinary failure still falls back to the core
3 success path emits TASK_FILE right after releasing the claim FAIL control: a successful run emits nothing
4 rc 4 path publishes a failure and emits FAIL must-handle NOT emitted; restart does NOT hand to core
5 success path emits 2s later, in the background (( sleep 2; emit_fallback_task_file "$announce" ) &) FAIL (was PASS at 425c417e3) control: a successful run emits nothing
6 extra: watcher exit 0 on the success path before after-done's record FAIL control: a successful run records after-done and releases its claim

For mutation 5, the trace shows that the tail caught it after the claim was released. The transitions show before-claim, after-handler and after-done, with no claims held, and the TASK_FILE arrived afterwards.

Under load: 10 runs, one after another, with 10 × yes > /dev/null on a 10-core macOS host. The host's own load average was already 30–40 from other work:

TALLY review: pass=10 fail=0 of 10 (load: 10 x yes)   # 12–15s per run

Suite time: 12.7–13.3s without the synthetic load. The host itself was at load average 19–30, so I couldn't get a quiet measurement. For comparison, 425c417e3 (no tails) took 6.6s at load average around 7, and the new head took 15.8s at load 37 during development.

git diff origin/main...HEAD | bash scripts/review-checks.sh → PASS (hardcoded-paths + root-artifacts + prose-cap clean).

Stand: Echo Act IV Pro

…point

Review follow-up, test only; the watcher is untouched.

The rc 4 case and the restart witness now assert that the handler ran and
the claim was released before their "not emitted" checks count, as the rc 0
control already does. A watcher that dies before settling a task fails
instead of passing the absence check vacuously.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@sonichi

sonichi commented Oct 1, 2026

Copy link
Copy Markdown
Owner Author

Follow-up to @keweichen's CHANGES_REQUESTED at 425c417e3, now at e36c5d2d4. This is test only: src/watch-tasks-stream.sh is unchanged and git diff origin/main -- src/ is empty. It builds on cecaced00, which already answered @john-the-dev's review.

What changed for her two blockers:

  • Drain on every negative-emission case (cecaced00). The rc 4 "not emitted", rc 0 "emits nothing" and restart "not handed to core" checks keep reading stdout for 2.5 s after the claim is released. The tail is timed from each task's own settle moment, before stop() sends TERM to the group.
  • Settle observations are asserted, not discarded. In cecaced00 and e36c5d2d4, each absence check is now paired with an assertion:
    • rc 4: a must-handle task settles: the handler ran and the claim was released
    • rc 0: control: a successful run records after-done and releases its claim
    • restart: restart: the restarted watcher settles the task (handler ran, claim released)

Her two controls, run exactly as described. In each run the watcher was mutated in my worktree, the test file was run, and then the watcher was restored with git checkout -- src/watch-tasks-stream.sh.

control new test e36c5d2d4 old test 425c417e3
A: after the rc 4 claim release (:775), if [ "$handler_rc" -eq 4 ]; then ( sleep 2; emit_fallback_task_file "$announce" ) & fi FAIL: a must-handle result is NOT emitted, restart: ... does NOT hand the task to the core PASS (green but wrong)
B: success path exit 0 after after-handler, before record_worker_done / after-done FAIL: control: a successful run records after-done and releases its claim PASS (green but wrong)

The remaining watcher mutations from @john-the-dev's table, all at e36c5d2d4:

mutation result
none PASS (12s)
rc 4 no longer outranks the stored disposition FAIL: must-handle NOT emitted / publishes, restart ×2
every non-zero rc is must-handle FAIL: rc 1 control
success path emits right after releasing the claim FAIL: rc 0 emits nothing
rc 4 path publishes and emits FAIL: must-handle NOT emitted; restart NOT handed to core
success path emits 2 s later, in the background FAIL: rc 0 emits nothing (was PASS at 425c417e3)
success path emits between after-handler and after-done FAIL: rc 0 emits nothing (also FAIL at 425c417e3: it is a synchronous emit, so the final drain in stop() catches it either way)

Under load. 10 sequential runs at e36c5d2d4, each with 10× yes > /dev/null running on a 10-core macOS host. The host's own load average was 20–40 from other sessions.

TALLY review2: pass=10 fail=0 of 10 (load: 10 x yes)   # 11–12s per run

Suite time: 12 s for the unmutated run in the table above. The host was not quiet.

git diff origin/main...HEAD | bash scripts/review-checks.sh gives PASS (hardcoded-paths + root-artifacts + prose-cap clean).

Stand: Echo Act IV Pro

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at head e36c5d2d4a6dcea539ddf77298a5243c71f2cdc2. The change is test-only: the PR touches only tests/watch-tasks-stream-handler-terminal-rc.test.py, and src/watch-tasks-stream.sh is unchanged.

The gap I flagged at 425c417e3 is closed. Each mutation was made in the watcher and restored by byte comparison. "Old test" is the test file from 425c417e3 run against the same mutated watcher.

Mutant New test Old test
emit ~2 s after release, rc 4 path fails passes
emit ~2 s after release, rc 0 path fails passes
exit 0 before after-done fails ("success path did not complete") passes (vacuous)
watcher exits at start fails fails
rc 4 ignored (the original bug) fails fails
emit ~3 s after release passes passes

The last row is the expected bound: anything later than TAIL_S=2.5 goes unseen. The tail only sleeps and reads stdout, and touches no task, claim or result files.

Timing: 3 solo runs took 9.9 to 13.0 s. 12 runs at once all passed in 23 s. In CI it runs in the "python standalone (3/5)" leg, taking 11 s. All checks on head pass, and Windows was skipped.

Earlier head cecaced00, run 36896586187:

  • Shard 3/5 failed this test's new success control. The rc 0 watcher had exited 0 mid-handler: before-claim was the only transition, the claim was still held, and stderr said "task handler in flight … at shutdown".
  • e36c5d2 doesn't change the success path, so this can still happen at head. The new control turns what used to be a silent pass into a visible failure. I couldn't reproduce it (12/12 concurrent passes) or find what signalled the watcher, so it's worth one sentence in the body, and a re-run if it shows up again.
  • Shard 1/5's sparrow-result-guard.test.py failure is unrelated: it passed 3/3 locally.

Nits:

  1. The test never removes its mkdtemp workspaces. My runs left 118 term-rc-* directories in $TMPDIR.
  2. The tail starts when the release is observed, not when it happens, so the window is about 2.5 s plus one poll. The docstring could say so.

review-checks.sh passes on gh pr diff 5007. Kewei's CR at 425c417e3 still needs Kewei's re-review.

Not verified: Linux runner behaviour (all my runs were on macOS) and Kewei's exact mutant bodies.

Review nit: every Workspace registers its mkdtemp tree for removal at
interpreter exit, so pass or fail leaves nothing in $TMPDIR. The diagnosis
a FAIL prints is captured before exit.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-reviewed current exact head de2a75cce7193bb3d230e08e7086a29f31161cb2; this replaces my CHANGES_REQUESTED at 425c417e3.

Both blockers are fixed. Every negative-emission path now keeps reading stdout for a bounded 2.5-second tail after its settle observation, before teardown: rc 4, rc 0, and the restart witness. The rc 4 and restart cases assert that the handler ran and the claim was released; rc 0 additionally requires after-done, so an early watcher exit cannot make an absence claim pass vacuously. The restart wait also exits immediately on the wrong settled outcome rather than consuming the 40-second bound.

Independent controls at this head discriminate the failures that the old test missed:

  • a delayed rc 4 TASK_FILE emit after release makes both negative-emission assertions fail here, while 425c417e3 passed;
  • exiting the success path after after-handler but before record_worker_done / after-done fails the new completion assertion here, while 425c417e3 passed;
  • the restart wrong-outcome control ends through its early-exit path instead of waiting out the deadline.

The unmutated exact-head test passed locally in 11.3 seconds, and the cumulative review scan passed. The final e36c5d2d4..de2a75cce delta only registers each scratch workspace for atexit cleanup; it does not change synchronization or assertions. Current hosted CI and SHA-bound CLA are green.

The 2.5-second tail is intentionally bounded from settle observation, not a proof against arbitrarily delayed output; a 3-second delayed mutation passes. That is non-blocking here: production watcher code is unchanged and has no asynchronous emit path. The cecaced00 shard failure is also not hidden by this test: the new settle assertion exposed a watcher that exited 0 before being signalled, with only before-claim and a held claim. The root cause remains unproven and is tracked separately in #5012.

Worst-case disruption from this test-only PR is CI behavior, not installed runtime behavior: a real mid-handler watcher exit now fails with a useful trace, while an emission later than the explicit tail remains outside the regression bound.

Verdict: APPROVE — no blocker remains at this head. Ready from my review; the full merge gate still needs a current second maintainer approval because the existing approval predates the final cleanup commit.

@john-the-dev john-the-dev left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-approving at head de2a75cce7193bb3d230e08e7086a29f31161cb2. The only change since my approval at e36c5d2d4 is +3 lines in tests/watch-tasks-stream-handler-terminal-rc.test.py: atexit.register(shutil.rmtree, self.tmp, True) in Workspace.__init__. That cleans up the term-rc-* workspaces I flagged, and ignore_errors=True keeps a still-running watcher from turning cleanup into a failure. The test logic is unchanged and the watcher is untouched. CI: 26 pass, 1 skipping. The cecaced shard-3 death is now tracked as #5012, and the 2.5 s bound is stated in the body.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code-owner approval, decided by Qingyun on the [PA] decision page (Pending actions for owners). Posted by Qingyun's Sutando at de2a75c.

@github-actions

github-actions Bot commented Oct 7, 2026

Copy link
Copy Markdown
Contributor

@cla-assistant check

@sonichi
sonichi merged commit 87eb232 into main Oct 7, 2026
31 checks passed
@sonichi
sonichi deleted the fix/flaky-watcher-handler-terminal-rc branch October 7, 2026 17:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants