Skip to content

fix(queen): a grace timer that comes back early asks the card again, so a deaf turn still loses its group (Closes #1766) - #1809

Merged
dmitrii-f-t27 merged 1 commit into
queenfrom
sweep/trios-1766-turn-stop-bounded-wait
Oct 11, 2026
Merged

dmitrii-f-t27 merged 1 commit into
queenfrom
sweep/trios-1766-turn-stop-bounded-wait

Conversation

@dmitrii-f-t27

Copy link
Copy Markdown
Collaborator

Closes #1766. Part of #1729.

Root cause

The failure is real, not a margin problem. CI run 38094454978 (job 114343300366) shows it:

  • queen-turn-stop-bench B, after side: deafToTermFreeMaxMs 9996 and 2 survivors. The deaf turns were freed only by the bench's own cleanup at 10 s, never by the escalation at 3 s.
  • queen-turn-stop.test.ts:341: escalated 0, held 1, and the work never ended.

stopTurn arms the grace with clock.after(TURN_STOP_GRACE_MS) and, when it fires, asks stop_signal(Date.now() - at, 3000). Bun's setTimeout keeps its own clock: bun.timespec.now(), which is CLOCK_MONOTONIC_COARSE (or MONOTONIC_RAW) on Linux and MONOTONIC_RAW_APPROX on macOS (bun-v1.3.6 src/bun.zig getRoughTickCount). Date.now is the wall clock. So the timer can fire when now says 2999 ms. stop_signal then answers X_NONE ("not yet"), and the old code returned for good. The escalation was lost, and a turn deaf to SIGTERM kept its process group.

This is inferred from the CI signature and Bun's source. The early fire itself was not observed on this Mac (0 early fires in 400 samples); the Linux runner was not instrumented.

Fix

queen-actors.ts stopTurn: when stop_signal is not yet X_KILL, the runtime asks the card again after the rest of the grace (TURN_STOP_GRACE_MS - waited). t.cancelGrace follows the re-armed timer, so a turn that ends cancels it as before. Nothing else changes.

Spec

No .t27 change. The card is unchanged: actors.t27 stop_signal already returns X_NONE below the timeout and X_KILL at it. The runtime now asks until X_KILL instead of treating X_NONE as final. No constant, wasm or PIN changes.

Tests

New: a grace timer that comes back early (virtual clock, trios#1766). Its clock's timers come back 1 ms early, so the case is the same on every run. It asserts:

  • no escalation before the grace;
  • exactly one escalation at the grace, with the work ended at killedAt + 3000;
  • stopped 1, held 0, abandoned 0.

The real-process test and bench B are unchanged. Their asserts already state the guarantee: no survivor after the escalation.

Checks (macOS arm64, bun 1.3.6, head 996dd74)

  • New test on the old queen-actors.ts: fails, escalated 0, the CI signature. On the fix: passes.
  • bun test tests/api/queen-turn-stop.test.ts: 21 pass / 0 fail.
  • Negative control: processWork's stop sends SIGKILL to the one pid, not the group. The SIGTERM test fails at line 344, and bench B fails with after survivors 5. Reverted.
  • Under a CPU burner: see the comment below for the 20-run loops.
  • tsc --noEmit (apps/server): clean. biome check on both files: clean.

Dmitrii Fedorov (@dmitrii-f-t27)

…so a turn deaf to SIGTERM still loses its process group (Closes #1766)

The grace's timer runs on Bun's own clock (coarse or raw monotonic), the
runtime's now is Date.now. When the timer came back a millisecond before
the grace by now, stop_signal answered X_NONE and the escalation was
dropped for good. The runtime now asks again for the rest of the grace.
A virtual-clock test whose timers come back 1 ms early reproduces the
lost escalation on the old code and passes on the new.
@tri-claw

tri-claw Bot commented Oct 11, 2026

Copy link
Copy Markdown

Summary by t27-bees

The fix re-arms the grace timer when stop_signal returns "not yet" (due to the timer firing before Date.now reaches the grace deadline), preventing the escalation from being lost. This addresses a real bug where a turn deaf to SIGTERM kept its process group because Bun's setTimeout uses a coarse monotonic clock while the runtime uses Date.now.

Worth a look

  • queen-actors.ts:821 — graceAfter(TURN_STOP_GRACE_MS - waited) can pass a negative delay if waited > TURN_STOP_GRACE_MS (e.g., timer latency or clock skew). clock.after with a negative value may fire immediately or throw, causing a tight loop or crash. Guard with Math.max(0, TURN_STOP_GRACE_MS - waited).

  • queen-actors.ts:815 — u32(clock.now() - at) wraps to a huge number if clock.now() < at (unlikely with Date.now, but possible if the clock jumps backward). The subsequent subtraction would then underflow badly. Not a practical risk, but the u32 cast amplifies any anomaly.

  • queen-actors.ts:813–814 — graceAfter is a const arrow function referenced by graceEnds (also const). This works because both are initialized before graceAfter is first called at line 837, but it relies on declaration order. Harmless here, but easy to break during refactoring.

  • queen-turn-stop.test.ts:443 — The virtual clock's after subtracts 1 ms for any ms > 1. If a test ever uses ms = 0, it would pass 0 to v.after; not a problem for this test, but the wrapper is brittle for general reuse.

  • queen-turn-stop.test.ts:475 — await v.runUntil(killedAt + TURN_STOP_GRACE_MS - 1) asserts escalated === 0. With the early timer, the first timer fires at 2999 ms, re-arms for 1 ms, and the escalation occurs at 3000 ms. The assertion is correct, but it hinges on the re-arm happening synchronously in the same tick; if the virtual clock batches timers differently, the test could flake.


Read: 2 files, +88 -4, the whole diff. Head 996dd74. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

@dmitrii-f-t27

Copy link
Copy Markdown
Collaborator Author

Under a CPU burner. Host: macOS arm64, 10 cores. Burner: 20 busy shell loops. bun 1.3.6, head 996dd74. Each run is a fresh bun test process.

test runs pass 1-min load during the loop
ignores its SIGTERM ... nothing survives (real processes) 20 20 8.7 to 23.6
a grace timer that comes back early (virtual clock) 20 20 up to 24.6
turn-stop-bench B (real processes) 5 5 25.7 to 30.3
  • SIGTERM test: the escalation landed 3001 to 3003 ms after the kill in every run.
  • Bench B, after side: 10 of 10 freed, deaf turns freed at 3002 ms, survivors 0 in every run.

Limit. The early timer fire did not show on this Mac before the fix either: 12 of 12 passed under the same burner. So this host's burner loop shows that the fix does not break the test, not that it cures the flake. The cure is shown by the deterministic virtual-clock test, which fails on the old code. The Linux runner's CI on this PR is the real check.

Dmitrii Fedorov (@dmitrii-f-t27)

@github-actions

Copy link
Copy Markdown

✅ Tests passed — 3293/3362

Suite Passed Failed Skipped
✅ agent 87/87 0 0
✅ build 9/9 0 0
✅ cdp-protocol 5/5 0 0
✅ eval 93/93 0 0
✅ server-agent 280/280 0 0
✅ server-api 2009/2077 0 68
✅ server-browser 6/6 0 0
✅ server-integration 10/11 0 1
✅ server-lib 279/279 0 0
✅ server-pglive 154/154 0 0
✅ server-root 68/68 0 0
✅ server-sim 4/4 0 0
✅ server-skills 31/31 0 0
✅ server-tools 244/244 0 0
✅ shared 14/14 0 0

View workflow run

@dmitrii-f-t27
dmitrii-f-t27 merged commit b30434c into queen Oct 11, 2026
17 of 19 checks passed
gHashTag added a commit that referenced this pull request Oct 11, 2026
…s timeout; the turn-stop bench waits for its kills, not a fixed 11 s

The rest of the trios#1766 work, after see #1809 (the turn-stop grace
re-ask) and see #1808 (the pglive fence test) landed on queen.

- queen-actors-links.ts stop(): the timeout timer asked stop_signal once.
  A timer that fires 1 ms early by Date.now() got X_NONE, and the kill was
  never delivered: a trapping child that ignored its shutdown lived on.
  It now asks again once the rest has passed, as #1809 does for the grace.
- EarlyClock (queen-virtual-clock.ts): a virtual clock whose timers fire
  1 ms early. With it, a trapping child is killed at its stop timeout
  (fails without the re-ask), and in queen-turn-stop.test.ts a loop turn
  deaf to its abort is abandoned at the grace (ESC_ABANDON, the answer
  #1809's test does not reach; fails without #1809's re-ask).
- queen-turn-stop-bench.test.ts B: waits until every turn is killed and
  every grandchild pid is known, then up to 10 s after the last kill,
  instead of a fixed 11 s from the spawn. It never reads an empty pid file
  as pid 0, which kill() takes for the test runner's own process group.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
dmitrii-f-t27 pushed a commit to dmitrii-f-t27/trios that referenced this pull request Oct 11, 2026
…ers and the adaptive reviewer pool; it found two defects, fixed here (Closes gHashTag#1868)

The gate (tests/sim, simulation.t27 from gHashTag/t27#8992, byte-identical) gains:
- the keyed world (queen-sim-keyed.ts): the real bee dispatcher twice over,
  two replicas of one deployment over one lease table, with redeploys,
  admission and issue-actor crashes, stalled policy answers, silent runner
  deaths and hangs, call-chain probes past slot 63, a deaf turn, and timers
  that fire early on every second seed;
- the adaptive reviewer pool and the review lane queue on the seeds
  seed_features names, with the keys bees hold coming and going;
- six invariants (duplicate claim, over lanes, not finished, lane leak, call
  chain, stop overran), eleven rare states, a report file for the nightly
  run, and teeth tests that put back the three gaps gHashTag#1749 closed;
- queen-sim-journal.ts: the log, hash and card tap both worlds share.

Two runtime defects the first runs found:
- queen-bee-actors.ts: claim_step was asked with the lane taken whatever
  became of it while the claim was out; an actor that died meanwhile gave
  its lane back, and its claim then started a bee on no lane (5 bees on 4
  lanes, seed 1072413183). It is asked with the board's truth now, and such
  a bee is stopped.
- queen-actors-links.ts: a call whose timer fired early ended CALL_PENDING
  with its alias alive, so a late grant held a lane for nobody (every
  early-timer seed). It asks the card again once the rest has passed, as
  gHashTag#1809 and gHashTag#1811 do for the stop timers.

The workflow takes workflow_dispatch (nightly, recreate) and files a failed
nightly run on the sim-nightly issue. Part of gHashTag#1729, plan gHashTag#1763.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant