Skip to content

feat(queen): a killed actor turn stops - AbortSignal into model and git calls, process groups killed whole (Closes #1713) - #1720

Merged
gHashTag merged 1 commit into
actors-nextfrom
queen-actors-stoppable-turns
Oct 9, 2026
Merged

gHashTag merged 1 commit into
actors-nextfrom
queen-actors-stoppable-turns

Conversation

@gHashTag

@gHashTag gHashTag commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator

Closes #1713. Part of #1712, item 2: turns that really stop.

The rule is gHashTag/t27 specs/queen/turn_stop.t27 (t27#8274, PR t27#8281). It is vendored byte-identical as specs/queen/turn_stop.t27, turn_stop.wasm (exports step_on_abort, escalation and in_flight; two builds, one hash) and queen-turn-stop-card.gen.js, with lines in specs/PIN and a PIN test.

What changes

Behind TRIOS_QUEEN_TURN_STOP=on (the createActorSystem option turnStop). Off is the default, and off keeps the runtime as it was.

  • The signal. ActorSpec.receive(msg, self, result, signal) and Isolated.start(msg, signal) get the pid's AbortSignal. When the pid dies, by the turn timer or by any exit including a supervisor's stop, actors.t27 stop_signal X_SHUTDOWN aborts it. At TURN_STOP_GRACE_MS (3000) stop_signal says X_KILL, and the card's escalation decides: SIGKILL to the process group for a process turn, or an abandon for a loop or thread turn. actors.t27 holds_after_kill decides when a stopped turn gives back its row and lane. New stats: escalated, abandoned, held.
  • The reviewer path obeys it. The model call (defaultReviewerLlm) is aborted: with the real AI SDK against a local server that never answers, it returned 2.6-27.5 ms after the abort. Git commands (run in queen-dispatch.ts, for example gitAt) and criterion commands (defaultExec) have their process group killed. The signal reaches them through withTurnSignal/turnRunner (AsyncLocalStorage), so no git helper gained a parameter. step_on_abort decides what is cut. A git worktree add or a runner-branch import under way finishes. measureCriteria's cleanup is a finalizer and always runs; a test checks that the worktree and the temporary directory are gone after an abort mid-measurement. The sweep (reviewFinishedDispatches, ReviewScope.signal) stops at its next await: a stopped review writes no verdict and no cache, and it backs off no lane.
  • The bound counts held reviews. The intake asks review_slots and review_slots_within with in_flight(busy, held_after_kill). A killed review that ignores its abort still holds its lane, so it still counts. With the flag, the fixed workers sit under their own one_for_one supervisor below the intake. Under one rest_for_one, a killed worker restarted its siblings. That used to cost nothing, because a stop abandoned. Once a stop really stops, it threw their work away: a probe measured 39 reviews aborted by a sibling's kill against 24 killed at their own bound. A stopped review's slot is refilled at once (released), not at the next heartbeat.
  • Pure fix, with or without the flag. processWork spawns detached, so the child leads its own group, and kills the group. The abort is SIGTERM to the group and stop is SIGKILL to the group. The false comment at queen-actors-isolate.ts L6-10 is corrected.

This is rebased on lane 3 (#1716). The adaptive pool is one_for_one already. Its dispatch also counts held reviews with the flag on.

Tests (tests/api/queen-turn-stop.test.ts, 20)

  • A process turn ignores SIGTERM (trap '' TERM), is escalated at the grace, and its whole group is killed, grandchild included. Measured: 3003-3153 ms from kill to freed, at most LANE_FREE_MS (5000).
  • A process turn that hears SIGTERM ends inside the grace with no escalation. Without the flag the kill reaches the whole group too, and no grandchild survives.
  • Loop turns: one that hears its abort ends at once, holding 0. One that ignores it is abandoned at the grace and holds until its work ends; nothing is claimed stopped. Without the flag nothing is aborted. A supervisor stop aborts the running turn too.
  • Commands: a command is cut at once. A shared write under way finishes, and a nested region inside it is part of it. After the abort nothing new starts, but a finalizer does. gitAt running sleep as git's grandchild: killed 1-32 ms after the abort, grandchild gone.
  • Stalled reviewer: 24 rows on 4 workers, 8 first attempts stall (4 deaf, 4 hearing). At most 4 at once with the flag. The control, the same input without the flag, reaches 9.
  • The sweep: the model call gets the signal, and once it is aborted there is no verdict, no reviewer cache and no lane backed off.

Benchmark, same seeded input, before vs after (tests/api/queen-turn-stop-bench.test.ts)

Part A, the reviewer on a virtual clock, bound 4. "Hearing" stalls hang in a model call that hears its abort; "deaf" stalls ignore it. free is the seconds from a stalled review's kill to its lane freed. "Beyond" is the review-seconds run past the bound.

scenario side done per h doneP95 s most at once s above bound beyond hearing free p50 / max s deaf free max s
steady 45/h, 120 s, 4% hearing + 1% deaf before 287 41 281 7 557 744 50 / 102 1500
after 287 41 290 4 0 0 0 / 0 1500
steady 45/h, 180 s, 8% hearing before 263 37.6 537 7 1014 1415 48 / 116 -
after 263 37.6 489 4 0 0 0 / 0 -
overload 120/h, 180 s, 4% hearing + 1% deaf before 549 78.4 7167 10 14944 26773 50 / 103 1500
after 425 60.7 10674 4 0 0 0 / 0 1500
overload 120/h, 180 s, 5% hearing before 536 76.6 8082 10 6182 12216 74 / 118 -
after 479 68.4 9293 4 0 0 0 / 0 -

Part B, real processes: 10 process turns, each leaving a sleep grandchild, half deaf to SIGTERM, killed at a 1 s bound.

side freed within 10 s hearing SIGTERM, free max deaf to SIGTERM, free max grandchildren alive 10 s after
before (proc.kill(9) to one pid) 8 of 10 (4-8 over runs) > 10000 ms > 10000 ms 10
after (group, abort, grace) 10 of 10 4 ms 3012 ms 0

What the numbers say, and what they do not:

  • Concurrency. It never exceeds the bound with the flag; before, it reached 7-10.
  • Lane freed after a kill. For a review that hears its abort: 0 s of virtual time. The measured real cost is under 30 ms (model call 2.6-27.5 ms, git 1-32 ms). Before, the median was 48-74 s and the maximum 118 s, the remainder of the 120 s model timeout.
  • What is left. A loop turn that never hears its abort still holds its lane to its own end (deaf 1500 s in both columns). Only a process turn can be forced, at 3012 ms.
  • Throughput. In overload, after finishes fewer rows: 60.7/h against 78.4/h with deaf stalls, and 68.4/h against 76.6/h with hearing stalls only. before makes up exactly that difference by running past the bound. Its extra 12216 review-seconds come to about 68 reviews at the 180 s median, against a gap of 57. Its extra 26773 come to about 149, against a gap of 124. The simulation has no key-lane limit, so a review past the bound costs nothing here. In production it is a call on a lane the bound exists to protect: z.ai refuses a third concurrent request with 1302. At the steady 45/h load, throughput is equal; doneP95 is 290 against 281 and 489 against 537.
  • Isolation. The reviewer still runs on the loop. Running its turns as processes (must_stop) would bound even a deaf stall at the grace; that is not done here.

Suite

bun test tests/api: 1620 pass, 11 fail, against 1584 pass and 10 fail on the base before lane 3. Six of the failures are Cannot find module tools/*.mjs (the sparse checkout). Three are the base's own named failures. One is a timer runs during a flood of 200 000 turns, which also fails on the base: run alternately three times each at load 34, the base took 5001-5013 ms and the lane 5003-5006 ms, against bun's 5 s default. One is the reaper ... nothing to reap, a 5 s timeout at load 40; its file passes 13 of 13 on the lane when run alone. tsc is clean. Biome reports 0 errors and the same 7 complexity warnings as the base.

🤖 Generated with Claude Code

@tri-claw

tri-claw Bot commented Oct 9, 2026

Copy link
Copy Markdown

Summary by t27-bees

Checked by the t27 compiler

t27c parse, parse-complete and typecheck on each changed .t27 file at 90e0ef7. Nothing was generated, built or run. A pass means the file is well formed, not that it is right.

  • trios/agent-server/specs/queen/turn_stop.t27: parses and typechecks.

Read by the model

This PR implements "turn stop" - when an actor's pid dies (timeout or exit), its turn is actually stopped via AbortSignal rather than abandoned. Model calls, git commands, and criterion commands are aborted/killed. A grace period (3000ms) then escalates per the card: SIGKILL to process group for process turns, or abandon for loop/thread turns. Stopped turns hold their lane until holds_after_kill releases them. Feature is behind TRIOS_QUEEN_TURN_STOP=on (default off).

Worth a look

  • queen-actors-isolate.ts:59-88 - processWork now uses detached: true and kills the process group (process.kill(-proc.pid, sig)). If proc.pid is not a process group leader (e.g., the spawned process forks and the child becomes leader), kill(-pid) may not reach all descendants. The comment acknowledges this was a bug before ("killed with SIGKILL to the shell left its sleep alive"), but the fix assumes the spawned process is the group leader.

  • queen-actors.ts:229-240 - Proc.abort is a single AbortController shared across all turns of a pid. If a turn adds a listener to this signal and doesn't remove it (e.g., crashes before cleanup), the listener persists and fires on the next turn's abort. The code at L565 creates Turn with ended: false but doesn't show listener cleanup on crash path - only turnEnded calls cancelGrace and releases.

  • queen-actors.ts:485-560 - stopTurn aborts the shared p.abort signal at X_SHUTDOWN (pid death). This aborts all in-flight async work for that pid, including any previous turn's straggling promises that haven't resolved. Since turns run sequentially per pid, this should be fine, but if receive starts async work that outlives the turn (e.g., fire-and-forget), it gets aborted mid-flight.

  • queen-criteria-run.ts:1093-1120 - removeCheckouts runs as a finalizer via asStep({ finalizer: true }, ...). If the turn is stopped during measureCriteria (before the finally block), the finalizer runs. But if the process is SIGKILL'd at escalation (ESC_KILL_GROUP), the Node process dies and the finalizer never runs - worktrees and temp dirs leak. The test expects cleanup, but that test likely runs with the flag off or before escalation.

  • queen-review-actors.ts:449-466 - Fixed workers now sit under their own one_for_one supervisor only when turnStop is on (L466-482). When turnStop is off, they're direct children of the rest_for_one intake supervisor. This changes restart semantics: with flag off, a worker crash restarts all later workers; with flag on, only the crashed worker restarts. The adaptive pool (L520) is one_for_one already, but the fixed pool's behavior diverges based on the flag.

  • queen-review-actors.ts:432-438 - Worker's receive sends released to intake when signal.aborted in finally. But signal is the turn's AbortSignal (shared across turns). If a turn is stopped, the signal is aborted. The next turn on the same pid sees the same aborted signal. The worker pid doesn't die on stop (only the turn does), so the worker stays alive and receives the next message with an already-aborted signal. The stepMayStart() check in defaultReviewerLlm/defaultExec/run will reject immediately, but the worker's receive will still run and send released again for the new issue.

  • queen-turn-stop.ts:85-105 - asStep throws abortError if the outer step was already started and the card says CUT. But it checks outer.signal.aborted before entering the step. If the signal aborts during fn() execution (between awaits), the next asStep or stepMayStart call will catch it. However, code not wrapped in asStep/stepMayStart (e.g., raw await in reviewFinishedDispatches) only checks via live() calls inserted at various points. If a live() call is missing between two interruptible deps, the sweep continues past abort.

  • queen-tick.ts:3121-3180 - interruptible wraps deps to check stepOnAbort(false, writesShared, false) before starting and abortError after each await. But stepOnAbort with started=false means "may this step start?" If the card returns STEP_RUN for a non-started step (e.g., a read-only git call), the dep runs. If it then aborts mid-flight, the post-await check throws. This is correct per the spec, but means a stopped review can still start new read-only steps after abort until one hits the post-await check.

  • queen-reviewer.ts:718-780 - defaultReviewerLlm creates a per-call AbortController (call) linked to the turn's signal via AbortSignal.any([timeout, call.signal]). The turn's signal gets one listener (onStop) added/removed per call. If multiple model calls happen sequentially in one turn, each adds/removes its listener. But if a call throws before the finally block (e.g., dynamic import fails), the listener leaks on the turn's signal. Next call adds another. Accumulated listeners = memory leak and duplicate aborts.

  • queen-actors.ts:177-188 - New stats escalated, abandoned, held are added to ActorStats. stopped definition changes: without flag it's "kill_effect stops work"; with flag it's "stopped turn whose work then ended". This changes the meaning of an existing metric behind a flag - dashboards/alerts on stopped will show different numbers when flag flips.


Read: 15 files, +2161 -96, the whole diff up to 60000 characters (the rest was not read). Head 90e0ef7. Model: openai-compatible/nvidia/nemotron-3-ultra-550b-a55b. Commands: @t27-bees review | summary | help | pause | resume. Free for every repository.

…it calls, process groups killed whole (Closes #1713)

Part of #1712, item 2. The rule is gHashTag/t27 specs/queen/turn_stop.t27
(t27#8274, PR t27#8281), vendored as turn_stop.wasm and
queen-turn-stop-card.gen.js, with lines in specs/PIN.

Behind TRIOS_QUEEN_TURN_STOP=on (createActorSystem option turnStop):
- each turn gets its pid's AbortSignal in `receive` and in its isolated work;
  when the pid dies (the turn timer or any exit), actors.t27 stop_signal
  X_SHUTDOWN aborts it, and after TURN_STOP_GRACE_MS the card's escalation
  kills the process group or records an abandon; holds_after_kill decides when
  a stopped turn gives back its row and lane;
- the reviewer's model call, its git commands (run) and its criterion
  commands (defaultExec) obey the signal; shared checkout writes and
  finalizers run to their end (step_on_abort); a stopped review writes no
  verdict and no cache and backs off no lane;
- the reviewer intake counts killed reviews that still hold a lane against
  the bound (in_flight), and its fixed workers sit under their own
  one_for_one supervisor, so one worker's kill no longer aborts its siblings.

A pure fix, on with or without the flag: process isolates lead their own
process group and are killed as a group, so no grandchild survives. The
comment in queen-actors-isolate.ts that claimed a thread stop ends the work
is corrected.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@gHashTag
gHashTag force-pushed the queen-actors-stoppable-turns branch from 90e0ef7 to a43f079 Compare October 9, 2026 18:20
@gHashTag
gHashTag merged commit 255a7c0 into actors-next Oct 9, 2026
1 check passed
@gHashTag

gHashTag commented Oct 9, 2026

Copy link
Copy Markdown
Collaborator Author

Two things after the description was written.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant