Repository navigation
feat(queen): a killed actor turn stops - AbortSignal into model and git calls, process groups killed whole (Closes #1713) - #1720
Conversation
Summary by t27-beesChecked by the t27 compilert27c parse, parse-complete and typecheck on each changed .t27 file at
Read by the modelThis PR implements "turn stop" - when an actor's pid dies (timeout or exit), its turn is actually stopped via AbortSignal rather than abandoned. Model calls, git commands, and criterion commands are aborted/killed. A grace period (3000ms) then escalates per the card: SIGKILL to process group for process turns, or abandon for loop/thread turns. Stopped turns hold their lane until Worth a look
Read: 15 files, +2161 -96, the whole diff up to 60000 characters (the rest was not read). Head |
…it calls, process groups killed whole (Closes #1713) Part of #1712, item 2. The rule is gHashTag/t27 specs/queen/turn_stop.t27 (t27#8274, PR t27#8281), vendored as turn_stop.wasm and queen-turn-stop-card.gen.js, with lines in specs/PIN. Behind TRIOS_QUEEN_TURN_STOP=on (createActorSystem option turnStop): - each turn gets its pid's AbortSignal in `receive` and in its isolated work; when the pid dies (the turn timer or any exit), actors.t27 stop_signal X_SHUTDOWN aborts it, and after TURN_STOP_GRACE_MS the card's escalation kills the process group or records an abandon; holds_after_kill decides when a stopped turn gives back its row and lane; - the reviewer's model call, its git commands (run) and its criterion commands (defaultExec) obey the signal; shared checkout writes and finalizers run to their end (step_on_abort); a stopped review writes no verdict and no cache and backs off no lane; - the reviewer intake counts killed reviews that still hold a lane against the bound (in_flight), and its fixed workers sit under their own one_for_one supervisor, so one worker's kill no longer aborts its siblings. A pure fix, on with or without the flag: process isolates lead their own process group and are killed as a group, so no grandchild survives. The comment in queen-actors-isolate.ts that claimed a thread stop ends the work is corrected. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
90e0ef7 to
a43f079
Compare
|
Two things after the description was written.
|
Closes #1713. Part of #1712, item 2: turns that really stop.
The rule is gHashTag/t27
specs/queen/turn_stop.t27(t27#8274, PR t27#8281). It is vendored byte-identical asspecs/queen/turn_stop.t27,turn_stop.wasm(exportsstep_on_abort,escalationandin_flight; two builds, one hash) andqueen-turn-stop-card.gen.js, with lines inspecs/PINand a PIN test.What changes
Behind
TRIOS_QUEEN_TURN_STOP=on(thecreateActorSystemoptionturnStop). Off is the default, and off keeps the runtime as it was.ActorSpec.receive(msg, self, result, signal)andIsolated.start(msg, signal)get the pid'sAbortSignal. When the pid dies, by the turn timer or by anyexitincluding a supervisor's stop, actors.t27stop_signalX_SHUTDOWN aborts it. AtTURN_STOP_GRACE_MS(3000)stop_signalsays X_KILL, and the card'sescalationdecides: SIGKILL to the process group for a process turn, or an abandon for a loop or thread turn. actors.t27holds_after_killdecides when a stopped turn gives back its row and lane. New stats:escalated,abandoned,held.defaultReviewerLlm) is aborted: with the real AI SDK against a local server that never answers, it returned 2.6-27.5 ms after the abort. Git commands (runin queen-dispatch.ts, for examplegitAt) and criterion commands (defaultExec) have their process group killed. The signal reaches them throughwithTurnSignal/turnRunner(AsyncLocalStorage), so no git helper gained a parameter.step_on_abortdecides what is cut. Agit worktree addor a runner-branch import under way finishes.measureCriteria's cleanup is a finalizer and always runs; a test checks that the worktree and the temporary directory are gone after an abort mid-measurement. The sweep (reviewFinishedDispatches,ReviewScope.signal) stops at its next await: a stopped review writes no verdict and no cache, and it backs off no lane.review_slotsandreview_slots_withinwithin_flight(busy, held_after_kill). A killed review that ignores its abort still holds its lane, so it still counts. With the flag, the fixed workers sit under their own one_for_one supervisor below the intake. Under one rest_for_one, a killed worker restarted its siblings. That used to cost nothing, because a stop abandoned. Once a stop really stops, it threw their work away: a probe measured 39 reviews aborted by a sibling's kill against 24 killed at their own bound. A stopped review's slot is refilled at once (released), not at the next heartbeat.processWorkspawnsdetached, so the child leads its own group, and kills the group. The abort is SIGTERM to the group andstopis SIGKILL to the group. The false comment atqueen-actors-isolate.tsL6-10 is corrected.This is rebased on lane 3 (#1716). The adaptive pool is one_for_one already. Its dispatch also counts held reviews with the flag on.
Tests (
tests/api/queen-turn-stop.test.ts, 20)trap '' TERM), is escalated at the grace, and its whole group is killed, grandchild included. Measured: 3003-3153 ms from kill to freed, at mostLANE_FREE_MS(5000).gitAtrunningsleepas git's grandchild: killed 1-32 ms after the abort, grandchild gone.Benchmark, same seeded input, before vs after (
tests/api/queen-turn-stop-bench.test.ts)Part A, the reviewer on a virtual clock, bound 4. "Hearing" stalls hang in a model call that hears its abort; "deaf" stalls ignore it.
freeis the seconds from a stalled review's kill to its lane freed. "Beyond" is the review-seconds run past the bound.Part B, real processes: 10 process turns, each leaving a
sleepgrandchild, half deaf to SIGTERM, killed at a 1 s bound.proc.kill(9)to one pid)What the numbers say, and what they do not:
afterfinishes fewer rows: 60.7/h against 78.4/h with deaf stalls, and 68.4/h against 76.6/h with hearing stalls only.beforemakes up exactly that difference by running past the bound. Its extra 12216 review-seconds come to about 68 reviews at the 180 s median, against a gap of 57. Its extra 26773 come to about 149, against a gap of 124. The simulation has no key-lane limit, so a review past the bound costs nothing here. In production it is a call on a lane the bound exists to protect: z.ai refuses a third concurrent request with 1302. At the steady 45/h load, throughput is equal; doneP95 is 290 against 281 and 489 against 537.must_stop) would bound even a deaf stall at the grace; that is not done here.Suite
bun test tests/api: 1620 pass, 11 fail, against 1584 pass and 10 fail on the base before lane 3. Six of the failures areCannot find module tools/*.mjs(the sparse checkout). Three are the base's own named failures. One isa timer runs during a flood of 200 000 turns, which also fails on the base: run alternately three times each at load 34, the base took 5001-5013 ms and the lane 5003-5006 ms, against bun's 5 s default. One isthe reaper ... nothing to reap, a 5 s timeout at load 40; its file passes 13 of 13 on the lane when run alone. tsc is clean. Biome reports 0 errors and the same 7 complexity warnings as the base.🤖 Generated with Claude Code