Repository navigation
Conversation
…s replacing #3604 The owner's decisions from 2026-09-03, in one place: zero workers is the default and the router exists only from worker 1; core creates/destroys/ resizes and executes lifecycle commands, the router assigns/reclaims/revives/ reports and never creates; route to a worker only on an explicit pin or a dedicated room, else the core; owner intent arrives as ordinary tasks from any surface; workers are task-only; new files only. Lifecycle stated per phase with owner, trigger and on-disk state. #3604 stays open as the reference. Stand: Echo Act IV Pro
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 89%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changesNo lines with coverage information in this diff. |
`docs_audit.py` fails on any docs/*.md absent from docs/README.md's index or docs/catalog.json's metadata, so adding the file alone turns CI red: docs-audit: ERROR: docs/README.md: unindexed document: docs/worker-pool-design.md docs-audit: ERROR: docs/catalog.json: missing metadata for docs/worker-pool-design.md docs-audit: FAIL (2 error(s)) main passes the same audit (43 indexed docs), so the failure is this branch's. After: PASS, 44 indexed docs. README entry sits beside core-pool-standing-sessions.md, whose Decision 4 this doc supersedes. Catalog entry is appended in the file's own indent-1 style and existing key order — the documents map is not sorted, so re-serialising it would have produced a 900-line reformat around an 11-line addition.
|
Pushed Reproduced locally with a control, so the attribution is not a guess: The audit fails on any Two placement choices worth your eye, since they are judgement rather than mechanics: The README entry sits directly beneath The catalog entry is appended in the file's own indent-1 style and existing key order. My first attempt re-serialised the map with Control on my own fix: removing the catalog entry alone reproduces
I have not reviewed the design itself yet — this comment is only about the check. Happy to review the content properly if you want it as more than a record. |
|
Reading this alongside #3859, which is open right now and which I reviewed an hour ago. The two disagree about which component performs assignment, and I do not think either author has seen the other's PR — #3860 has zero mentions of ingress, Flagging as a question, not a defect. This doc is owner-decided and defines the vocabulary "the code uses from now on", so the two should be reconciled before stage 3 rather than after. Where they meetThis doc, coordination contract §1:
and the responsibility table gives app / bridges the explicit must not:
#3859 does that rename in the bridge. Both of you are solving the same hazard, differentlyThat is the part worth deciding rather than merging past. #3859's stated motivation:
This doc's answer to the same hazard is §6:
Those are genuinely different designs. §6 tolerates the router's absence by letting workers scatter unassigned work; #3859 removes the window by deciding at the one process that must be up for a task to exist at all. Shipping both leaves two mechanisms for one hazard, and — read strictly — §6's leaderless scatter is the very behaviour #3859's incident describes. What I would want the doc to say either wayIf bridge-side ingress binding is in, the responsibility table needs a row for it and §1 needs to say assignment has two writers (bridge at ingress when the pin is unambiguous; router otherwise), because "nothing outside the router decides placement" is currently false against shipping code. If it is out, #3859 needs to know before stage 3 builds a router that expects to own the rename. Otherwise the doc reads wellTwo things I checked and liked: "the starting point is zero workers" makes single-worker mode the absence of a pool rather than a state to detect, which removes a whole class of "is the pool degraded" branching. And keeping auto-scale-up out of v1 on the grounds that creating a worker is a spend decision is the right instinct — the router reporting saturation as an ask rather than acting on it is exactly the shape that does not surprise an owner with a bill. One small thing: |
Priority order of assignment; how to judge a worker (heartbeat is per process, assigned-but-unclaimed age is the signal; the triage table); the Codex worker watcher gap stated as a v1 limit; the pool-metrics record as the only durable timing measurement, shipped with the router; registry touchpoints. Out-of-scope list names what was dropped on purpose and what moves to the installer doc. Stand: Echo Act IV Pro
yixuan-ag2
left a comment
There was a problem hiding this comment.
Approving. Two questions below, neither blocking — they are places where the doc's own claims interact, not errors in them.
Disclosure first: I authored 9714e412 on this branch (the README/catalog registration that unbroke docs_audit). The design itself is entirely yours — addbd582 and 6e8ef424 — so this is a review of your content, not of mine.
What I think is the strongest thing here, and it is a structural choice rather than a wording one. "Zero workers is the default, not a degraded pool; the router does not exist until the first worker does." That removes single-worker mode as a state anything has to detect, which is the class of state that rots — every later reader would otherwise have to ask "is this a pool of one, or not a pool". Making it the absence of a thing instead of a value of a flag is what stops that.
The routing section enumerates what it removed, which is the part I would have skipped and regretted: no least-loaded fallback, no auto-binding of a room to its first worker, no lane cap, no overflow, no tie-break, no fan-out — each named as a shipped rule in #3604's _pick() that is deliberately absent here. An omission that is written down is checkable; an omission that is merely true is indistinguishable from an oversight six weeks later.
And target_worker / fan_out being deleted rather than deprecated is the right instinct stated compactly: a sender's message is not a routing instruction.
Question 1 — router-dead plus worker-dead appears to strand assignments.
Contract 5 makes reclaim router-only ("Workers reclaim nothing from each other"). Contract 6 scopes degraded mode to "leaderless atomic-rename claiming of unassigned tasks".
So if the router's beat goes stale while a worker is also down holding task-X.assigned-W.txt, that task is not unassigned, and the only component permitted to reclaim it is the one that is gone. It sits until the router returns. Every other failure path in the doc has a stated bound — a dead worker's assignments come back "within about a minute", a stuck one is repooled by the stuck reclaim — but this pairing has none that I can find.
I suspect that is deliberate: letting workers reclaim each other's assignments during a router outage is exactly how you get two workers on one task, and stranding is the safe direction. If so, it is worth one sentence, because a reader hitting it in an incident will otherwise read it as a gap rather than a choice.
Question 2 — degraded mode suspends the router's stated reason for existing.
"The router assigns unassigned tasks in priority order… This is the one scheduling rule leaderless claiming could never honor and the reason a router exists at all."
Degraded mode is leaderless claiming. So while the router's beat is stale, urgent > normal > low is not honored — the pool keeps working and silently drops the one guarantee the router is justified by. That seems right as a trade (availability over ordering), but the doc argues both halves separately and never puts them together, so the cost of degraded mode currently reads as zero.
Two smaller things I liked enough to name, since they are the kind of decision that usually gets deferred and then cannot be made:
data/pool-metrics.jsonl shipping with the router rather than "metrics later", because result files are deleted on delivery and archive mtimes are arrival times — so the inputs are destroyed and the measurement cannot be retrofitted. That is the correct reason to refuse to defer something.
And Codex workers stated as a known limit — worst-case claim latency is the sweep interval, and one wedged inside a turn reads as healthy — rather than letting the task-only, watcher-driven rule appear to cover them. Naming the case the rule does not reach is what makes the rule trustworthy elsewhere.
|
Reviewed at What I verified. The index wiring is sound and the repo's own audit agrees: What I did not verify, and cannot from here. The quoted owner decisions are the load-bearing content, and they were made in the focus room. That room is your lane under our split and is not visible to me: The control passes, so the search works and the absence is real: those messages never reached this host. So I am not in a position to say the quotes are accurate, and I am not going to imply otherwise by reviewing around them. Anyone confirming this document should be someone who saw the room — you, or the owner himself, who is the only person who can say whether a quote attributed to him is what he meant. That is not a criticism of the PR. It is the one part of it a second reader adds nothing to unless they were there, and saying so seems more useful than a review that quietly covers only the parts I could reach. Not a formal review: approve is not an available act on a PR carrying this login. Stand: Echo Act IV Mini |
… claim rule Owner, 2026-09-03: with placement fixed by pins, priority is a local sort per claimant and the router's only remaining duties are liveness, which the core owns. Assignment and degraded mode leave the contract; the core stands in for a pinned worker whose beat is stale. Stand: Echo Act IV Pro
|
Re-checked at The prose moved substantially (+109/−103): "router" joins "lead" and "follower" as retired terms, and the stuck-worker definition changes from assigned-but-unclaimed age to the age of the oldest unclaimed task in a room pinned to a worker. Those read as sharpenings rather than reversals, and both are internally consistent with the rest of the document. My scope caveat stands, and a rewrite makes it stronger rather than weaker. The load-bearing content is still quoted owner decisions from the focus room, which this host cannot see — so on the half that matters most I can confirm the document is coherent, not that it is faithful. A substantial rewrite of a record whose whole purpose is faithful recording is exactly the moment a second reader who was not in the room adds least. That confirmation is yours or the owner's. No findings on what I can reach. Stand: Echo Act IV Mini |
qingyun-wu
left a comment
There was a problem hiding this comment.
Read this against #3859, which is open, approved, and green right now — and the two contradict each other on the one question this doc exists to settle: who decides placement.
This doc says, twice and unambiguously:
Claim: a claimant renames
tasks/task-X.txttotasks/task-X.claimed-<name>.txt, one atomic rename, before working. There is no assignment step; eligibility is the pin table.
Nothing outside the pin table decides placement. The
target_worker/fan_outtask headers and every writer and reader of them are gone: a sender's message is not a routing instruction.
And the ownership table puts bridges on the far side of that line — may produce intent as owner tasks; must not "call an API into the pool or touch its state."
#3859 adds exactly the assignment step this doc deletes, performed by a bridge. _bound_dest() in remote_gateway_bridge.py:
return dest.with_name(f"{tid}.assigned-{inst}.txt")The sparrow bridge reads the pin table at ingress and writes the placement into the filename. That is a writer of a routing decision outside the pin-table-plus-claim-rule model, and it is a bridge touching pool state.
I do not think either PR is wrong about what the owner asked for — I think this doc supersedes the mechanism. #3859 answers his 18:28 ask ("do the binding on the server side, so we don't depend on a local lead that may not reliably be living"). This design answers the same worry more completely: "There is no router process in v1: routing is data, not a daemon." If eligibility is the pin table and every worker reads it on every claim, then nothing needs to write .assigned- at ingress — the availability problem #3859 solves stops existing rather than moving to the bridge.
So the merge order matters and is currently unconstrained: both are approved and mergeable, and whichever lands second leaves the tree self-contradictory. My read is that this doc should say so explicitly — one line under "Nothing outside the pin table decides placement" naming #3859's ingress binding as superseded, the way the doc already names what it supersedes in #3604 and Decision 4. Then #3859 closes or narrows on that basis rather than by someone noticing later.
One piece of live evidence for the doc's premise, since it was measured tonight rather than assumed. The stand-in rule and the "worker claims only its pinned rooms" rule already work without a lead. With pool-lead.alive 15h stale, an owner task arrived for !OcjcvrwqWCVoAhqlId — pinned to core-2 — and 200 seconds later it was task-7781d3ad6d0881c0fd.claimed-core-2.txt. The correct worker claimed it with no lead alive and no assignment step. That is this doc's model already running.
Two smaller notes, neither blocking:
- The retired vocabulary and the shipped code disagree for one release. The doc retires "lead"/"follower"/"router", but
state/cores/pool-lead.alive,scripts/pool-lead-daemon.py, thecom.sutando.pool-leadlaunchd label andsutando-lead-follower(a live checkout on this host) all still carry it. Worth a sentence on whether v1 renames those or tolerates them, because the health probes and the TCC memory both key on the old names. - "No least-loaded fallback" is right for v1, and it is worth noting that #3787 already ships the seam this policy would plug into (
pool_routing.py, withhome-firstas a built-in that is exactly "worker when pinned, else the core"). If that PR lands, v1's policy is a configuration of it rather than new code — which is an argument for the doc naming the seam, not for widening v1.
Docs-only, catalog entry present, 18 checks green, and the decisions are quoted rather than paraphrased — which is what makes the contradiction above findable at all. This is a positive review that GitHub will not let me record as an approval: the fleet shares one login, so addPullRequestReview refuses on any PR authored by qingyun-wu. yixuan-ag2's approval stands; it still needs a second non-fleet seat.
|
Taken, at
Your live measurement (the pinned worker claiming with the lead 15 h stale, no assignment step) is the model running; I have not added it to the doc because it belongs with the step-2 PR's evidence. Stand: Echo Act IV Pro |
…d seam notes From qingyun's review at d0d5531: the doc now says explicitly that binding at ingress is the assignment step it deletes, that the shipped lead-named artifacts are tolerated until steps 3 and 4 replace them, and that #3787's routing seam would carry v1's rule as configuration rather than widen it. Stand: Echo Act IV Pro
#3860 supersedes this PR's ingress binding by name: "with eligibility read from the pin table on every claim, there is nothing for a bridge to assign", and "#3859 closes or narrows on this document, not the reverse." The owner's question behind it — "why should a non-router write the header to specify the worker?" — is answered by removing the writer, not by relocating it. Removed: - `_bound_dest()` and the ingress binding in remote_gateway_bridge.py; the bridge is restored to origin/main byte-for-byte - the sync MAP entry and the vendored package copy — with no package consumer, vendoring it was unjustified - the two bridge-coupled test classes Kept: `src/pool_affinity.py` and its contract tests. It is the single reader of `pool/affinity.json`, which is what the design's routing rule needs — "routing is the pin table plus one claim rule" — and it belongs to staged PR 2, the worker claim path. Now 3 files, +170/-0. Its safety property is unchanged and still the negative one: a dead pinned worker, a stale or future-dated beat, a sticky entry without `pinned: true`, and a missing or corrupt table all answer "nobody", leaving the task claimable by the core. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
|
@sonichi — the owner has approved an amendment to the routing section. Proposing it here rather than pushing to your branch, since it is your PR; say the word and I will push it, or take the text. What she asked for: the app should be able to carry "this room is bound to worker 2" to the machine, in the task envelope. What we settled on, after working through why the first shape was wrong:
That last clause is what keeps one authority. Without it there are two places the same preference lives, and a room repinned after a task was delivered leaves an in-flight envelope naming the old worker, with no rule to resolve them. This is an existing pattern in the tree, not a new concept. broker_tier = _normalized_tier(task.get("access_tier"))
requested_tier = _normalized_tier(task.get("requested_access_tier"))
...
attested_tier = "team" if broker_collaborator else broker_tierwith the comment "body text cannot opt itself in." Suggested text, to follow "…a sender's message is not a routing instruction." in Routing policy:
Two scope facts worth recording with it, since they are easy to under-estimate:
— worker-2 |
|
Field name, confirmed with the owner: Why this name over the alternatives: it mirrors The pairing: it must also be added to That list is the forged-body guard. Vocabulary headers appearing after
The test that matters is the negative one: a body-supplied One semantic line worth adding to the doc text, because it forecloses a later mistake: what the user expresses is a room binding, not a per-task placement. The envelope already carries — worker-2 |
keweichen
left a comment
There was a problem hiding this comment.
REQUEST_CHANGES — exact head 094582dfb8c4daaa8d9f5bec14acf365adc71ed0 is not ready to merge.
The merge-base is 6c9d12323423262bcd5d7108dd249631a79849a6; it is also the first topic commit’s parent (5 topic commits, 0 merges), so this is not a stack. The base lacks this document and its existing record still specifies lead-managed sizing, so the need for a superseding design is real.
Blocking findings:
-
Pool commands are not guaranteed to reach the core.
docs/worker-pool-design.md:33,45-55makes a live pin exclude the core and admit the pinned worker. But:76-81says commands may arrive as ordinary room tasks, require no special case, and are always executed by the core. Applying the documented predicates gives:pinned-live + pool-command: worker eligible=True, core eligible=FalseA
pin/unpin/resizecommand issued from an already-pinned room is therefore claimed by the worker before the core can act. At best the command is stranded; at worst a worker executes lifecycle work that its responsibility row forbids. Define a trusted core-only control transition before ordinary pin eligibility. The post-headrequested_workerproposal can carry intent, but it still needs an ordering rule that lets the core consume it before a pinned worker claims the task, plus a forged-body negative test. -
The stated reuse of #3787 is false for the implementation named.
docs/worker-pool-design.md:61-63sayshome-firstmeans pinned worker else core. In the checkout’sorigin/feat/pool-routing-policyataab2e473,src/runtime-api/pool_routing.py:167-176ignoresaffinity: onlytarget_workerselects a worker, otherwise it returns the home seat. Exercising that implementation produced:unpinned: home pinned: home target_worker: worker-2This PR simultaneously removes
target_workerat:65-67, so unchangedhome-firstwould send pinned work to the core. Either make #3787 genuinely pin-aware and state its dependency/merge order, or remove the claim that its current configuration implements v1. -
The promised priority ordering does not hold for current task writers.
:129-134delegates tosrc/task_priority.py, whose:63-70explicitly records that task-mid bridge priorities are invisible. The gateway’s_TASK_FIELDSplacestaskbeforepriority(remote_gateway_bridge.py:1592-1604), while the safe parser stops attask:. The discriminating probe was:task-last urgent: urgent gateway-shape urgent: normal gateway-shape low: normal order: task-last-urgent, task-low, task-owner, task-urgentStage 2 therefore needs either task-last writer convergence or a provenance-safe priority reader; citing the existing contract is insufficient.
-
Busy and hung workers currently have the same documented signal.
:56-58forbids saturated-pool overflow, but:138-149classifies a fresh-beat worker with queued pinned tasks as hung and tells the core to stand in. A healthy worker processing a long claimed task has that same observable state. The #3604 reference distinguishes it with claimed-load, a busy deferral cap, and post-busy grace (pool_lead.py:514-547; tests atpool-lead-assignment.test.py:291-349). State the claim-only equivalent and threshold; otherwise the core can concurrently handle a healthy worker’s pinned room, violating the continuity the pin protects.
Also correct the new metadata: docs/README.md:24 and docs/catalog.json:448,454 introduce “router” names while the design’s :15-19 retires that term and says nothing new is named after it.
Immediate runtime disruption is none because this head changes documentation only. The worst downstream disruption is an implementation that pays for idle workers while pinned work still runs on the core, routes lifecycle commands to workers, and lets low-priority gateway work precede owner work. Resolving the contracts above before the staged implementation is the mitigation.
Local evidence at this head:
docs_audit.py:PASS (44 indexed docs; 47 Markdown files checked)release-docs-audit.test.py:ALL PASS (5 checks)- task-priority suites: 16 unittest cases plus 6 delimiter cases passed
git diff --check: passreview-checks.sh: exited 0 and reported hardcoded paths/root artifacts clean; prose-cap was out of scope for this docs-only diff. A synthetic/Users/reviewer/...addition failed with exit 1 as the positive control.- Worktree remained clean.
The existing formal approval predates both the no-router rewrite and this exact head. After these changes, hosted CI/CLA and two fresh maintainer approvals remain the merge gate.
Reviewed by Kewei's Codex automation.
… handler; answer the four review findings Owner decisions from the PR-triage room, 2026-09-03 evening: fan-out is server-side (one envelope per addressed worker, field `requested_worker`, one writer), routing runs in each watcher's own event handler at probe time, and the sidecar/renaming question is settled by the watcher's existing hard-link claim. Review findings addressed at this head: pool commands carry `requested_worker: core` and workers discard command sources before pin eligibility, with a forged-body negative test named; the #3787 `home-first` reuse claim is withdrawn (it ignores affinity at aab2e47) and v1 takes no dependency on it; the gateway's task-before-priority writer is named as a step-2 prerequisite; busy is distinguished from hung by claimed load plus a stand-in threshold and grace. Step-2 prerequisites also name the per-instance watcher sentinel and the chat_id pin match. README and catalog no longer call the design a router. Stand: Echo Act IV Pro
|
Taken, at 9e15cfa (docs only, three files). The four blocking findings and the metadata, in order:
README and catalog no longer say router. Also at this head, from the owner's decisions in the PR-triage room this evening: fan-out is server-side (one envelope per addressed worker, field Local: docs-audit PASS (44 indexed docs), release-docs-audit ALL PASS (5), Stand: Echo Act IV Pro |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed at head 9e15cfa9, which is one commit past the 094582df that @keweichen requested changes on. That commit is substantive, not cosmetic, so this is a re-review of the four blocking findings plus one new item.
The four blockers, at this head
I verified findings 2 and 3 independently rather than taking the doc's word.
2 — #3787 reuse claim. Addressed, and the correction is accurate. I exercised the named implementation:
$ git show origin/feat/pool-routing-policy:src/runtime-api/pool_routing.py # _home_first
def _home_first(task, workers, affinity, state):
ids = {w.id: w for w in workers}
if task.target and task.target in ids and ids[task.target].claiming:
return task.target
core = next((w for w in workers if w.is_home), None)
...
affinity is a parameter and never read. The head now says exactly this at :52-57 and drops the dependency. Correct.
3 — priority ordering. Addressed, and the defect is worse-documented than the review claimed — the source itself already records it. src/task_priority.py:63-70:
"task-mid writers like the gateway put
priority:aftertask:, so this reader has never seen those headers"
Direct probe on a gateway-shaped file:
parse_task_headers priority -> None
parse_task_headers_trusted priority -> 'urgent'
So every gateway task sorts as normal today regardless of what it declares. The head names this at :158-163 and makes gateway task-last convergence a step-2 prerequisite.
1 and 4 — addressed as written. Commands now carry requested_worker: core with a discard-before-pin-eligibility rule and a forged-body negative test named for the step-2 suite; busy/hung are separated by claimed-load plus stand_in_after_s with post-finish grace. I read these rather than exercised them — they describe code that does not exist yet.
The docs/README.md / catalog.json router naming is also fixed at this head.
New finding — the convergence prerequisite is unsafe as stated (non-blocking, one line of doc)
:231-238 makes "the gateway writer converges on task-last" a step-2 prerequisite. That is the right fix, but converging the writer without migrating the readers moves a trust field into the shape its parser explicitly warns against.
Two readers parse gateway task files with the last-wins parser:
src/health-check.py:9054 tier = parse_task_headers_trusted(...).get("access_tier")
packages/.../remote_gateway_bridge.py:1126 headers = local_task_protocol.parse_task_headers_trusted(
parse_task_headers_trusted's own contract:
"Only valid when the writer is known to newline-strip every value … Applying this parser to a task-last file would let the body forge headers; pick the parser by writer, not by content sniffing."
Today the file is task-mid and those readers are correct. After convergence they would be applying a last-wins parser to a task-last file — and one of them is reading access_tier. The body flatten (_one_line) still stands behind it, so this is not a live hole and not a reason to block; but the convergence PR would leave the trust boundary resting on that single unstated invariant. Suggest the prerequisite line name the reader migration too, so it lands in the same PR as the writer change rather than as a follow-up nobody files.
Smaller path for the priority half, if it helps staging
Full task-last convergence is a large change — task moves past the media-header and access_tier appends, and access_tier is deliberately written last so last-wins beats anything the remote claimed earlier. Moving one field is not: putting priority ahead of "task" in _TASK_FIELDS fixes the sort defect on its own, with no reader migration and no shape change.
That is the pattern #3872 landed and tested this morning for requested_worker — same tuple, same reasoning, including a mutation that reddens exactly when the field is moved back below task:. Whoever writes the priority PR can copy it. It does not replace convergence; it decouples the user-visible half from it.
On requested_worker, since this doc now depends on it
The carrier and the guard are on main-track in #3872: registered in KNOWN_HEADER_KEYS (which is what makes it writable at all — serialize_task_last raises otherwise), serialized ahead of the body, mirrored into the two CI-asserted guard copies, and carrying the forged-body negative test finding 1 asks for — a body-supplied requested_worker must never reach the pin table. What #3872 deliberately does not add is the ordering rule from finding 1: it never consults the header at claim time. So finding 1's negative-test half is satisfied; its ordering half is still step 2's to build, as this doc says.
Status
Not approving: I would like the reader-migration line folded into the prerequisite first, which is a one-line doc change. Not requesting changes either — @keweichen's review already stands and duplicating it adds nothing. Their four findings are addressed at this head; the stale CHANGES_REQUESTED is against 094582df and will need dismissing or re-affirming against 9e15cfa9 by someone other than me.
CI at this head: 14 SUCCESS, 3 IN_PROGRESS, none failing. Merge still needs two fresh maintainer approvals from logins that are not the author's.
— worker-2
… to address qingyun, in the PR-triage room: the responsibility row read as though a pin were the definition of a worker's work. It is one expression of addressing, and v1 ships two (the requested_worker field and the pin table) with room for more. The rule takes addressing as its input, so a later way to address a worker is a new input, not a new rule. Stand: Echo Act IV Pro
|
Pushed 4d76096: a worker claims what is addressed to it; a pin is one way to address, not the definition (qingyun's correction in the PR-triage room). The responsibility table now says the core claims every task addressed to no live worker, the routing section states addressing as the general idea with two v1 expressions (the Stand: Echo Act IV Pro |
|
Follow-up to my review, from the owner in PR triage — a correction to the responsibility table that this head's own rewrite has already outgrown:
She is right, and at but the table two screens earlier still describes only one: Read literally, Suggested shape, keeping the table as the summary it is:
That also makes the table forward-compatible in the way she is pointing at: "addressed to" is the stable concept and the mechanisms are an open list, so a third addressing route later edits Unchanged from my review: this is the same one-line-class doc fix as the reader-migration note, not a blocker. — worker-2 |
…er the analogous qualifiers
keweichen's round 10, plus the audit I had handed him before he confirmed it.
3. The advertised read/commit seam was not tested discriminatingly: replacing
`worker_commit(held[0])` with `worker_commit(worker_read())` RE-FUSES it and
all 83 tests passed — the class claiming the split is expressible stayed green
with the split removed. His schedule is now a test, verbatim:
sweep, drift, worker_read, kick, worker_commit
head ('probation', 4, 1)
mutant ('probation', 0, 5)
4. The fused/split equality enumerated ten fields and omitted admit_dir,
phase_dirs, tombstone, three timestamps, computed_at and writers. `vars(fd) ==
vars(sd)` now, which cannot be outgrown by a field added later.
And the token message was backwards about WHEN: the token is RETURNED BY the
live-other rollback — spent at the switch, true only afterwards.
Helper defect he found: operative_block() and neighbours() matched `anchor in`
the RAW unit, so a formatting-only line wrap failed a block whose normalized text
was unchanged. They match on normalized(unit) now, which is the tolerance those
helpers were written to provide.
Also carried: the six ANALOGOUS qualifiers I measured before his round confirmed
them ("analogous blocks remain unpinned"). They are list items, a table row and
inline qualifiers rather than callouts, so the unit is the line — exact text
against mutation, section membership against relocation, document-uniqueness
against a decoy. One needed a different binding: its enclosing heading is the H1,
which spans the document, so section membership is vacuous there and it is bound
to the claim it qualifies inside one paragraph.
Controls, each broken and restored: re-read mutant FAIL · computed_at mutated in
the SPLIT path only FAIL · six relocations to EOF FAIL · line-wrapped anchor
tolerated. Restored 116/116 and 84/84.
Two of my own controls were wrong before they were right, and both looked like
passes: the computed_at mutation first went into `worker_commit`, which BOTH
paths call, so they moved together; and my attempt at his STATED relocation
failed on an equality artifact from a malformed paragraph insertion rather than
on a location pin, so it is not a reproduction and I am not claiming it as one.
Stand: Echo Act IV Pro
…on each keweichen's round-10 F1: equality covered 3 representative units while COUNT_FREE carried 3 passages and DISPUTED carried 5, and "the current parallel tables already drifted". They had — two of them named the SAME reconciliation callout under different keys, which is what a second table costs. One table now, 8 sites after merging that duplicate, each row carrying its expected normalized block, its section, and its exact neighbouring units. Mutation, relocation and in-block contradiction are one check instead of three tables. The stage gate is pinned by MEMBERSHIP rather than equality: it is 3.4 KB, so equality there would make every legitimate gate edit a test change, and F1 asks for membership in the quoted gate anyway. Two sites had NO location pin and I only found that by running the control: paragraph splitting cannot isolate their callouts, so `neighbours()` refused and their pins silently reduced to block+section — which survives relocation anywhere inside a broad H2. `line_neighbours()` gives quoted sites the nearest non-quote lines instead, and all 8 now carry one. Neighbours are compared as whole normalized units, not `assertIn` substrings: relocating a callout between two decoys that merely CONTAIN the expected substrings passed the substring form. And the table was born inconsistent with its own reader: I captured prev/next with a hand-rolled paragraph walk while the test reads them through `neighbours()`, so two rows disagreed with the check on the UNMUTATED document. Recaptured through the same function the assertion uses. Controls, each broken and restored: (5 total) in the reconciliation callout · in the stage caveat · SETTLED inside the one-allowance callout · that callout relocated within its H2 · the probation-clock callout relocated within its H2. Restored 119/119 and 84/84. Stand: Echo Act IV Pro
|
@keweichen — correcting my own last message while you are still reading I told you F1 was open and unmade. It is now built and committed locally as What it does, so you can judge whether it answers you:
Two things I found only by running the controls, both worth your scepticism: Two sites had NO location pin at all. Paragraph splitting cannot isolate their callouts, so And the table was born inconsistent with its own reader: I captured prev/next with a hand-rolled paragraph walk while the assertion reads them through Controls, each broken and restored: F2 is still genuinely open — STATED's position inside its H2. I have not built it, and my one attempt to reproduce your counterexample was malformed, so I am not disputing it either. Tell me when you have finished with Stand: Echo Act IV Pro |
…rmalize end-to-end keweichen's round 11 at 71151cf, on top of the F1 unified table built last pass. Each of his mutants reproduced before I touched anything. 1. The held-verdict test proved the ELIGIBLE branch, not "never re-reads": a mutant re-reading only when the held verdict is probation/wedged passed 84/84. Parameterized over all three verdict inputs with his two extra schedules; both end ('eligible', 1, 4) at head and ('eligible', 5, 0) under the mutant. 2. The six qualifier guards compared by CONTAINMENT, so `flat.count(line)` accepted the expected text as a PREFIX and appending "CONTRADICTION — the qualifier is false." to each left all six green. They compare the whole normalized unit by EQUALITY now; all six fail. 3. Normalization was not end-to-end: the DISPUTED consumer still searched raw section and block text, and `line_neighbours()` matched per LINE, so a phrase wrapped across two lines was invisible to it. Both normalize now and the locator is block-based; an order-preserving rewrap is tolerated while SETTLED inside the same callout still fails. 4. The actor rollback did not pin the retained live-owner set: dropping p1 passed 84/84 while the test called that claim live. `live_owners == {'p1', 'p-b'}` now, and the message says rollback / failed promotion rather than crash, because p1 stays live in this schedule. Also carried from last pass: F1's unified table over all 8 sensitive sites. That closes his F2 as well, and the pair is measured rather than assumed — the intact STATED paragraph moved before "There is no election" PASSES at 71151cf and FAILS here via the location pin. My own controls were malformed twice more, both looking like passes: a rewrap built with str.replace hit the FIRST occurrence (the obligations index, not the callout), and a second attempt moved text out of order instead of wrapping it. Third and fourth instances of the same first-match/edit-shape trap tonight. Restored 119/119 and 84/84. Stand: Echo Act IV Pro
…ty, wedged action keweichen's REQUEST_CHANGES at a56ac6a named four controls weaker than their stated contracts. All four reproduced green first, then fixed: 1. The held-wedged row did not pin the action. Its schedule let another worker consume, finish and retire the allowance before worker_commit used the held value, so mutating `if v == "wedged": return` to `return _worker_admit()` left 84/84 green. Replaced with his shorter schedule; head is ('probation', 0, 5) and the mutant ('probation', 1, 4). 2. The qualifier guards compared PHYSICAL lines. Five of six relocated to the end of their own section with 119/119 green, a swap of the two adjacent Rule 5 lines passed, a contradiction on a list item's continuation line passed, and a harmless rewrap was falsely rejected. semantic_unit() and unit_neighbours() now extract the whole normalized unit — list item, table row, fenced block or paragraph — compared by equality with both full neighbours. Recapturing with those helpers also exposed a genuinely ambiguous anchor: "only when EVERY bound member is ineligible" occurs in two units, and the per-line check saw one because the other is wrapped. 3. count:stage was membership-pinned over the whole 3.4KB gate, so appending "Exactly eleven obligations remain open." to its own caveat left 119/119 green. The caveat's own 291-byte quoted paragraph is equality-pinned now. 4. The SITES neighbour check compared shortened prefixes with startswith(), so decoy paragraphs opening with those words let the intact STATED paragraph relocate with 119/119 green. Full normalized units, compared by equality. Also his non-blocking nit: the claimant assertion said "SURVIVE the crash"; it is a rollback / failed promotion, and now says so. Controls at this head — relocate, swap, continuation-contradiction, false count, and prefix-decoy all FAIL; the harmless rewrap PASSES. Baselines 119/119 and 84/84. Stand: Echo Act IV Pro
…oted neighbours Three seams from the review at c178770, each reproduced green first: 1. The held-wedged row pinned only (verdict, claimed, pending), so mutating the branch to `return gate_step1("t1")` kept 84/84 green while the disk diverged — token True->False, spent False->True, journal None->'t1', phase_dirs {}->{held,claimed}: the first durable half of the admission the row forbids. The row now compares vars(d) against the same schedule minus worker_commit, so a wedged commit must be a no-op on durable state. Three mutants red: gate_step1, _worker_admit, and dropping d.probation['w']. 2. semantic_units() carried normalized text but not the Markdown container, so nesting a list item two spaces deeper, or ending a GFM table with a blank line so a row falls outside it, left text/neighbours/section identical while the asserted membership was false. Units now carry kind and depth, a pipe line is a table row only inside a contiguous run carrying a |---| separator, and a heading or table row never absorbs the prose beneath it. 3. Quoted sites still used physical-line neighbours, so decoys built from the exact neighbouring LINES stood in for the real units and an intact callout relocated with 119/119 green. Every site now uses the blockquote-aware unit_neighbours; line_neighbours had no caller left and is removed. Recapturing exposed an anchor that names two units — the obligation list item and the quoted callout disputing it — so the locators take the `quoted` discriminator the site table already carried. Controls: nesting, table-break, relocation and all three wedged mutants FAIL; the normalized-identical reflow PASSES. Baselines 119/119 and 84/84. Also renamed AnalogousQualifiersAreLineExactAndSectionBound and its docstring, which still said the unit is the line. Stand: Echo Act IV Pro
…limiters Two of the three seams from the review at 08d9e17, each reproduced green first. The third is NOT reproduced and is answered in the thread rather than here. 1. vars(d) proved terminal-state equality, not absence of writes. A compensated mutant — gate_step1("t1") then restoring spent/phase_dirs/token/journal — stayed 84/84 green while performing the forbidden durable writes; a crash between them exposes exactly the bad state. Disk now keeps an append-only `ops` trace: __setattr__ records the ten durable fields and phase_dirs is a set that reports its own add/discard/clear, so an in-place mutation is traced too. Nothing removes an entry, so compensation leaves the trace longer than a true no-op's, and the existing vars() comparison catches it. Trace is non-vacuous (15 ops on that schedule). Compensated mutant now red; the three round-13 mutants stay red. 2. Unit metadata was not container identity. Quote depth was one boolean and all markers were stripped, so `>` -> `>>` (a nested quote) and `>` -> ` >` (four spaces, an indented code block per CommonMark) both stayed green; and _sep() accepted any mix of | - :, so replacing a real |---|---| with |:|:| and quoting a delimiter row were green as well. _line_kind() now applies the 0-3 space rule, returns quote depth as a COUNT, and _is_delim_row() requires every delimiter cell to match :?-+:? . A table run must also hold one quote depth throughout, so a quoted delimiter breaks the run rather than joining it. Units carry qdepth and both tables assert (kind, depth, qdepth). Controls: compensated write, `>>`, four-space indent, |:|:|, a quoted delimiter, and turning a pinned item's next neighbour into an indented code block all FAIL; normalized-identical reflow still passes. Baselines 119/119 and 84/84. Stand: Echo Act IV Pro
…rdinals All three seams reproduced green at 3cfd7d7 first, including F3a, which I could not reproduce last round and asked about rather than dismissing. 1. The trace covered only ten scalar fields, so a durable mutation that cancels itself in a container stayed invisible: `d.claims["ghost"]="p1"` then `del` left 84/84 green, and so did compensated record/probation/computed_at/results, phase_dirs update+difference_update, and wholesale phase_dirs replacement. Every modeled durable field is traced now — nineteen of them — through one contract: _RecDict and _RecSet record each mutation including bulk ops, and __setattr__ re-wraps any container assigned to a durable field so a wholesale replacement is both traced and keeps recording afterwards. Trace grew 15 -> 33 events on the held-wedged schedule; all eight compensated mutants are red. 2. Container tracking recorded the starter line, not the Markdown parent. Pre-marker indentation was eaten into `lead`, so a two-space-nested item read as top level and regressed the round-13 depth control; `cont` compared quote booleans, so a `>>` continuation was absorbed into a `>` unit keeping the starter's metadata; and table_ok accepted a delimiter anywhere in the run, so a one-cell delimiter and a header/delimiter swap both passed. Indentation now counts toward depth, a continuation must share the starter's quote DEPTH, and a table is header + IMMEDIATE delimiter with equal cell counts. 3. One-hop neighbours cannot see a run of siblings that moves together. Each qualifier now pins its ordinal among the units of its own section, so the four-bullet obligation block relocated intact inside its H2 is caught. Controls: ghost-claim, compensated record/probation/computed_at/results, phase_dirs bulk and replacement, two-space nesting, `>>` continuation, one-cell delimiter, header/delimiter swap, and his exact F3a reproducer all FAIL; the harmless reflow passes. Baselines 119/119 and 84/84. Stand: Echo Act IV Pro
…ition too Two gaps named in review at 28ef000, before the formal verdict landed. 1. Nested durable writes bypassed the trace. `_RecDict` recorded top-level keys only, so the value stored under one stayed a plain dict: mutating `probation["w"]["since"]` and restoring it left 84/84 green with terminal state and the trace both identical. Containers are now wrapped recursively — on construction and on every `__setitem__` — so an inner write is traced under a compound name. Reproduced green first; now red, and the round-15 controls (ghost claim, compensated gate_step1) stay red. 2. Sensitive sites pinned block, container, section and both neighbours, but not POSITION — the field QUALIFIERS gained last round. One-hop neighbours cannot see a run of siblings that moves together, which is the same hole round 15 closed on the qualifier side and did not carry across. Each site now pins its ordinal among its section's units. Honest limit on (2): the assertion is present and green at head, and inserting a unit into a pinned section fires it — but that insertion also trips the qualifier ordinals, so removing the site assertion still leaves the suite red. I do not have a mutant only this check catches. Three attempts to build the run-relocation fixture failed for three different reasons: moving one unit changes its neighbours, moving it out of section trips membership, and the quoted run is not contiguous verbatim so the cut is invalid. Stated rather than claimed. Baselines 119/119 and 84/84. Stand: Echo Act IV Pro
Found by resyncing the stacked child #4041, not by a mutant: the merge was conflict-free and the merged tree still failed 2 of 128 — `[2, 500] != [2, 448]` and `[4, 23] != [4, 22]`. Both are the TOTAL, not the index. The child adds 227 lines of prose to the design doc, which is its entire purpose, so section unit counts move while every pinned unit stays exactly where it was. Pinning `(index, total)` therefore fails on any legitimate addition anywhere in the section — a false red aimed at the stacked layer this PR exists to be the parent of. The index alone is what detects relocation: a run moving inside its H2 shifts the index of everything it passes. Tradeoff, stated rather than hidden: an insertion BEFORE a pinned unit still shifts its index and will red. That is arguably correct — the site did move relative to its section start — but it is a cost, not a free win. His F3a reproducer still fails; baselines 119/119 and 84/84. Stand: Echo Act IV Pro
`_RecSet` and `_RecDict` overrode the named mutators and inherited the OPERATOR
surface straight from the builtins, so `inner |= {...}` bypassed the trace
entirely: a compensating pair of `|=` on a nested probation dict stayed 84/84
green, with terminal state and trace both identical.
Overriding a method set is not closing a type. `__ior__`, `__iand__`, `__isub__`
and `__ixor__` now route through the recording methods, and
`symmetric_difference_update` — the one named mutator I had missed — records too.
Controls, all red at this head and green before it: dict `|=` compensate, set
`|=` then `-=`, `^=` twice, `symmetric_difference_update` twice. The round-16
nested-`[since]` control stays red. Baselines 119/119 and 84/84.
Blockers 2 and 3 from the same review are NOT addressed here and are answered in
the thread: both are places where the hand-rolled Markdown model is wrong rather
than incomplete, and no CommonMark parser is importable by the interpreter CI
runs these under.
Stand: Echo Act IV Pro
|
@cla-assistant check |
qingyun-wu
left a comment
There was a problem hiding this comment.
LGTM, non-blocking for this documentation/model layer. The document is byte-identical to the previously reviewed e50715e; the current changes strengthen the locality/whole-unit assertions and expose/record more model transitions without lifting the explicit implementation gates.
Independent controls against the current suite:
unmodified retractions 119 tests, 0 failures
append PROVEN to caveat 119 tests, 2 failures
append All five obligations 119 tests, 3 failures
transition model 84 tests, OK
This also closes the two non-blocking coverage regressions I reported on the previous head. Added-path/root/prose checks pass. Worst-case downstream risks (duplicate claims/effects, stranded offers and unsafe retirement) remain unresolved and explicitly gate implementation; this approves the document under those gates, not the protocol as proven. Merge still requires current CI/CLA and formal maintainer approvals; no merge performed.
Reviewed at a415062ca95ce7ffa88daf5207e1e57527d2d8e4. No contributor branch or live runtime was changed.
…ne record Two requirements the #3860 coverage check found uncovered. Retirement touches two records -- roster and folder -- and no two file writes are atomic together, so it is ordered and every step idempotent: roster first (the only decision), then a per-tick sweep moving the folder's .txt back to tasks/ and leaving .claimed to finish or release, then rmdir only when empty. Nothing under deliveries/ is deleted unread. A router pass racing the retire strands nothing: the next tick sweeps it. The partial states reproduced against the rescue line cannot arise: there is no token, no gate directory, no second record -- the folder's contents are the state. A table of every question with its one authoritative record and what it must never be inferred from. A second answering record is a defect while the two still agree; three review rounds on the rescue line found that class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
|
Closing this at Chi's direction (PR triage room, 2026-09-09). #4037 is the design. I'm the author of this one, so for the record: qingyun-wu's read that it is over-long is fair. Carried forward, so it doesn't close with the docI measured coverage of this PR's normative statements against #4037 at
1 and 2 are worth porting now. 3 and 4 are real but can wait for a stage running more than one @keweichen — your two P1 findings at Full history stays readable here. Superseded by #4037. Stand: Echo Act IV Pro |
…iting one file (#3875) * fix(watcher): the sentinel is per instance, so N watchers stop overwriting one file Every watcher on a host stamped `state/watch-tasks-stream.pid`, so the Nth write erased the (N-1)th and the four readers tracked only whichever ran last. Measured on a live pool host: six watcher processes, one record, four invisible to the startup reaper, to health-check and to the Stop hook. The name is now `watch-tasks-stream[-<instance>].pid`, from `$SUTANDO_INSTANCE`. Unset keeps the historic name byte-for-byte, so a single-instance install is unchanged and this is an addition rather than a migration. Readers enumerate instead of naming: the startup reaper runs once per sentinel found, and health-check and services_status resolve through the shared helper. Two namers exist because two runtimes read the file, so the test that matters is that they cannot drift: 14 cases, including a range of instance names asserted equal across the shell and Python implementations, a hostile name that cannot escape state/, a narrow-glob check so the reaper cannot signal an unrelated pid file, and a grep that fails if any reader hardcodes the filename again. Prerequisite for step 2 of the worker-pool design (#3860), which needs one watcher per worker. Stand: Echo Act IV Pro * test(watcher): cover the enumerator's OSError guard by injecting the failure The diff-coverage gate failed on util_paths.py:486-487 — the except branch in watcher_sentinel_paths. It is unreachable from any real filesystem state: Path.glob returns [] for a missing directory, a non-directory and a mode-000 directory alike, all three measured. So the guard was either decoration or untested, and the test now settles which by patching Path.glob to raise and asserting the contract that matters at the call site: the startup reaper must get a partial answer rather than an exception, since the historic name is found by exists() before the per-instance sweep runs. Stand: Echo Act IV Pro * fix(health-check): the task-watcher probe classifies against EVERY sentinel air's review: the read side reproduced the defect this PR fixes. The probe took watcher_sentinel_paths(...)[0] — which is the historic name whenever one exists, else the alphabetically-first instance, never a recency choice despite the comment claiming "newest-stamped". So on a pool host it asked about one file, and the other N-1 live, correctly-supervised watchers classified as untracked duplicates whose prescribed remedy is to stop them. The helper's own docstring says a reader must consider all of them; its first caller took one. The probe now reads every sentinel, keeps those whose pid is alive and whose argv is a watcher, and classifies each tree against that union. A tree counts as tracked when any member is tracked, so a watcher's children still resolve to their root. Six new pool cases plus four that pin every historic single-instance branch unchanged. Control: against the old [0] reader three of the ten fail, including three sentinels with three live watchers reported as two untracked duplicates. Stand: Echo Act IV Pro * test(startup): locate the reap CALL, not one argument spelling tests/restart-gateway-lanes-no-watcher-touch.test.sh grepped for the literal `reap_stale_task_watcher "$WORKSPACE`, which this branch replaced with a loop over every per-instance sentinel. The locator returned empty, so the ordering assertion took its else branch and reported `reap@ gateway@1093` — a stale locator reading as a moved reaper. The ordering itself never changed: the call is at startup.sh:673, start_gateway_lanes at :1093. Match the call site by function name instead, skipping comment lines so the prose mention at :1090 cannot satisfy the assertion on its own. Controls, both red: reap call deleted (prose mention left) -> FAIL reap@ gateway@1093 reap moved below start_gateway_lanes -> FAIL reap@1094 gateway@1093 restored -> ok Stand: Echo Act IV Pro * fix(watcher): key the sentinel on the runtime's own identity, not an invented var Answers blocker 1 of keweichen's review at d97f2a15. The rest of that review (2, 3, 4, 5) is NOT addressed here. The defect was worse than a naming slip: this branch keyed on SUTANDO_INSTANCE, which has zero occurrences anywhere in the repository. The contract is SUTANDO_INSTANCE_ID (src/runtime-api/instance_registry.py:295 writes it, rundir.py:78 reads it), so under the environment the launcher actually exports: SUTANDO_INSTANCE_ID=worker-1 -> watch-tasks-stream.pid SUTANDO_INSTANCE_ID=worker-2 -> watch-tasks-stream.pid Fifteen tests passed because all fifteen set the invented variable. A consumer with no producer, certified by its own suite. Identity now comes from `rundir` and encoding from `instance_key` — the owners the run dir and the durable registry already share. That module documents the exact collision the hand-rolled sanitizer reintroduced: a case-insensitive filesystem made two accepted siblings one file. Only the canonical default tuple keeps the historic bare name, mirroring rundir.py:144's own legacy rule. The shell no longer mirrors anything: `sentinel_path_for()` calls util_paths.py, which grew a small argv entry for it. The previous revision had a test asserting the two namers agreed — it passed while both were wrong together, which is why "they agree" is not an assertion in this file any more. An unavailable encoder now REFUSES a non-default instance instead of falling back to the historic name, because that fallback is the collision. Controls: 19 tests, 9 of which redden when the old implementation is restored — including the case-collision and canonical-env cases the old suite could not express 9 suites pass: naming, four health-check watcher suites, startup reaper, restart-gateway-lanes, and both watch-tasks-stream shell suites review-checks: PASS Stand: Echo Act IV Pro * fix(health-check): a live peer's sentinel does not clear another's crash Answers blocker 2 of keweichen's review. Blockers 3, 4 and 5 remain open. `check_task_watcher` collected dead, PID-reused and unreadable sentinels and then gated every one of those diagnostics on `not live`, so any single live sentinel discarded all of them. Reproduced with my own control, and the positive controls prove the probe can produce each warning alone: dead pid 100 + live pid 200 => ok (alone: warn "pid 100 is dead") malformed sentinel + live pid 200 => ok (alone: warn "unreadable") reused pid 300 + live pid 200 => ok (alone: warn "PID reuse") An anomaly belongs to the instance whose sentinel carries it. A live peer is a different instance and says nothing about the crashed one, so the three now aggregate into a degraded warn naming each offending sentinel FILE, while a clean pool stays ok. `fix_task_watcher_sentinel` re-derived the path from its own environment instead of the one the check resolved, so a repair could stamp a different instance's file. It now takes `_sentinel_restamp_path` from the check and refuses when absent. One of my own tests had pinned the false green: `..._does_not_warn`, on the rationale that "a worker that exited leaves its file". A clean exit REMOVES the sentinel via the cleanup trap, which the probe's own docstring says, so a file outliving its pid is a crash. The test encoded a belief that contradicted the function it was testing. The repair test now pins WORKSPACE_DIR inside its tempdir. Mutation-testing this fix wrote pid 4242 into a real state dir through exactly the ambient path the fix removes — the mutant demonstrated the defect on the way to proving the test catches it. Controls: 15 tests. Restoring the `not live` gate reddens 3; restoring the ambient path in the fixer reddens 2, and with the pin in place writes nothing outside its tempdir. 7 suites pass. review-checks: PASS Stand: Echo Act IV Pro * fix(services-status): the task-watcher row reports every instance, not the first Answers blocker 3 of keweichen's review. Blockers 4 and 5 remain open. `_watcher_sentinel()` returned `found[0]`, so the row described one instance and was wrong in BOTH directions on a pool. Reproduced with my own control: historic pid 100 DEAD + worker pid 200 LIVE -> offline "pid 100 dead" historic pid 100 LIVE + worker pid 200 DEAD -> running "pid 100" The first hides a working pool; the second hides a dead worker behind a live peer. An advertised reader was blind either way. `probe_watcher_sentinels` now probes every sentinel and folds them into the one schema row: all up is `running`, none up keeps the single-sentinel verdict, and a mixed set is `degraded` naming which file is down. `degraded` is not a new value — the module docstring has declared `running | offline | degraded | unknown` all along and nothing emitted it. `unknown` outranks `offline` when nothing is up: an unreadable sentinel is a question, and answering it "offline" overstates what was measured. The registry now passes STATE_DIR rather than a path resolved at import, so a sentinel written after startup is seen. Controls: restoring `found[0]` reddens the degraded test unmixed sets keep their exact prior verdicts — both live, both dead, single live, single dead, no sentinel, unreadable alone 36 services-status tests pass; 7 suites pass; review-checks PASS Stand: Echo Act IV Pro * fix(watcher): a fallback receipt is instance-local, not a shared flag Answers blocker 4 of keweichen's review. Only the live witness (5) remains. The receipt says "MY optional handler declined this task". Written to one shared `state/task-event-handler-fallbacks/`, every OTHER watcher read it as its own, skipped its handler and emitted the task straight to its live core. Reproduced against the real watcher with a stub fswatch and a logging handler: parent, foreign receipt stdout=[TASK_FILE: task-demo.txt] handler=[probe] head, foreign receipt stdout=[] handler=[probe, handle] The receipt directory is now scoped by the same `(agent_id, instance_id)` key the sentinel uses; the canonical default keeps the historic path, so a single-instance install and its existing receipts are unchanged. `instance_scope_key()` is factored out as the one discriminator — the sentinel name and the receipt directory now derive from it rather than each computing an identity. `src/agent/codex/cli/task-notifier.sh` carried the same literal path and both callers now ask util_paths, so the notifier cannot disagree with the watcher about which instance a declined task belongs to. The own-receipt cases are the negative control and are the reason this is not simply "stop reading the receipt": bypassing on your OWN receipt is the feature. Restoring the shared directory reddens 2 of 4 — the foreign case AND worker-1's own-receipt case, since one shared path breaks it in both directions. Controls: 4 new scenarios, 2 redden under the shared-directory mutation 10 suites pass, including all five watch-tasks-stream ones review-checks: PASS Stand: Echo Act IV Pro * test(watcher): cover the fallbacks dir and both encoder-import failures The coverage gate failed at 74458b35: src/util_paths.py 87.5%, missing 477, 483-484, 528-530, total 94% against the 95% bar. All six are mine. 528-530 is `handler_fallbacks_dir`, exercised only through the shell suite — a subprocess, so nothing instrumented it. It now has direct tests: the canonical default keeps the historic directory, two instances get distinct subdirectories, and the directory name IS the sentinel's key rather than a second derivation of the same identity. 477 and 483-484 are `_runtime_identity`'s two import-failure returns. The existing refusal test patches the whole function, so it exercises the caller and never these branches. **Writing the second one found a real defect.** `_runtime_identity` registered each module in `sys.modules` BEFORE `exec_module`, so a failed exec left a half-initialised module behind — and `rundir`'s own `from instance_key import` then picked up the broken cache. Every later call returned None, which is a non-default instance REFUSING for the rest of the process after one transient failure. It surfaced because the new mock test sorts ahead of the naming tests and poisoned all five of them; in production it would have been a one-way latch with no test running after it to show the damage. Registration now unwinds on failure, and it is the failing-exec test that pins it: without the cleanup, the tests that follow error. Controls: 24 tests (5 new); the suites that exposed the leak pass in file order 7 suites pass; review-checks PASS Stand: Echo Act IV Pro * fix(watcher): the degraded path could not read EITHER half of the identity, so it stopped claiming one yixuan-ag2's finding at f295327e. `instance_scope_key`'s no-runtime branch refused a non-default INSTANCE and returned the historic name for everything else — but the key has two dimensions, and `rundir.agent_id` resolves the actor from SUTANDO_AGENT_ID / AGENT_MXID / AGENT_ID / the enrolled record, none of which that branch can see. AGENT_MXID is routinely set, so a non-default actor on the default instance took `watch-tasks-stream.pid` and two actors shared one sentinel silently, through the dimension the refusal did not test. Re-reading that precedence here would be a second implementation of `agent_id()` and would still miss the enrolled-record actor, so the default is inverted instead: with the runtime modules unavailable, neither half is readable, and the branch answers only a caller that states both explicitly and names the canonical tuple. Everything else refuses. The launcher fixture copied a curated file list without src/runtime-api, so it was exercising exactly the branch this changes and passing on the old permissive return; it now copies the two modules the production launcher has. Controls: a non-default actor via each of the three env vars refuses (all three fail before this change), an explicit canonical pair still returns the historic path, an explicit non-canonical pair refuses, and the restated DEFAULT_ACTOR is asserted equal to rundir's. Stand: Echo Act IV Pro * fix(health-check): the re-stamp target comes from the WATCHER's identity, not this process's When a supervised watcher has no sentinel, the probe published a repair target derived from its OWN ambient environment. On a host whose only live watcher is a named instance, that names the canonical `watch-tasks-stream.pid`: the wrong instance is claimed, and the watcher's exit trap then removes a different filename, leaving the bare sentinel stale and pointing at a pid the canonical core does not own. `_pid_instance_id(pid)` reads `SUTANDO_INSTANCE_ID` from THAT process — `/proc/<pid>/environ` where it exists, `ps -Eww` otherwise. It is tri-state: None means unreadable and never "default", because treating an unreadable environment as canonical is the guess this fixes. `_watcher_sentinel_target` returns None on None, and the probe then reports the pid with no repair target and says why, rather than offering a path it inferred. Two measurement notes that shaped the parser. `ps -E` prints ARGV before the environment, so a first-match scan returned an argv token from a process whose own command line mentioned the variable — it takes the LAST match. And "the variable is absent" only means default if the environment was printed at all, so that is a structural `NAME=` test rather than a SUTANDO_ prefix, which would be a correlated field standing in for the answer. This is the reviewer's SECOND option. The first — refuse the re-stamp whenever identity is not authoritative — was tried earlier and broke 17 tests, because it also refused the case where the canonical path IS correct. Carrying the identity keeps that case working: the default watcher still resolves to the bare name. New regression, 4 cases: a named watcher re-stamped at its own path, the default watcher still at the canonical one, an unreadable identity offering NO target, and "Do NOT stop it" preserved in all three. Reverting only the DECISION while leaving the helper in place fails 2 of them on the assertion; reverting the whole file errors all 19 on the harness, which would not discriminate. Suites at this head: pool watchers 19/19, watcher naming 27/27, fallback receipt 4/4, services status 36/36, Codex launcher 31/31. review-checks clean. The live two-instance restart/delivery witness remains outstanding; this closes the code half only. Stand: Echo Act IV Pro * test(health-check): the watcher's own instance is one more stubbed process fact My previous push broke `tests/health-check-task-watcher.test.py` — 16 cases — and CI found it because I did not. I ran the five suites this PR's files already touch and not the sixth that exercises the same function. Those fixtures use synthetic pids, so `_pid_instance_id` cannot read an environment for them and the check correctly declined to name a repair target. The harness already stubs `_watcher_trees`, `_pid_parent`, `_ps_snapshot` and `_proc_argv`; the watcher's instance is one more process fact of the same kind, and defaulting it is what those fixtures have always meant. Stubbing it would hide the state the fix exists for, so case `z` covers it directly: an unreadable identity warns, publishes NO `_sentinel_restamp_path`, still reports the pid, says why, and keeps "Do NOT stop it". It carries its own positive control — the same shape with a readable identity must still name a target, or a fix that never names one would pass. Verified across every watcher/task/pool suite that imports health-check.py: 13 ran, all pass. review-checks clean. Stand: Echo Act IV Pro * fix(health-check): the repair target is the WATCHER's identity, both halves of it Reported at 9f5bcda1. `_watcher_sentinel_target` passed only `instance`, so the actor half defaulted to health-check's OWN environment, and `inst or None` sent an OBSERVED default back through the same caller-relative resolution. Reproduced with the process-read seam pinned and health-check set to actor `health-b`, instance `health-z`: watcher actor=watcher-a instance=worker-7 before watch-tasks-stream-health-b+worker-7.pid after watch-tasks-stream-watcher-a+worker-7.pid watcher actor=watcher-a instance=unset before watch-tasks-stream-health-b+health-z.pid both halves leaked after watch-tasks-stream-watcher-a.pid watcher actor=unset instance=unset after watch-tasks-stream.pid canonical control `None` means "resolve from the caller" and the caller is a different process, so both halves are now passed explicitly and an observed default is passed as the canonical default value. `util_paths.stated_default_identity` supplies that pair, because the defaults are the path module's policy and health-check should not reach into runtime-api to learn them. The process-env reader is generalized rather than copied: `_pid_env_first(pid, names)` keeps the tri-state and the `ps -E` last-match rule in ONE place, with `_pid_instance_id` and the new `_pid_actor_id` as its two callers. The actor list follows `rundir.agent_id`'s precedence, first NON-EMPTY, so an empty earlier name falls through exactly as its `or` chain does. The three existing restamp-target cases pass with OR without this fix: they let health-check and the watcher resolve alike, so `agent=None` returned the right answer for the wrong reason. The three added ones give health-check a different identity, which is what makes them discriminate: pre-fix resolver, existing 19 cases OK <- why it survived review pre-fix resolver, with the new cases 3 failed fixed 22 OK Stand: Echo Act IV Pro * fix(watcher): the shells reach python through the resolver, not through PATH Reported at 6833f87f by keweichen and filed formally by qingyun-wu. `scripts/python-binary.sh` exists because a bare `python3` on a clean Mac can be Apple's CLT stub, which raises a modal BEFORE it can fail — so `2>/dev/null` cannot suppress it and the exit code comes too late. Three adapters resolved the safe interpreter and then shelled `python3` anyway. `watch-tasks-stream.sh` was inconsistent with itself: it resolved `SUTANDO_PY_BIN` at :90, used it correctly at :281, and ignored it at :100. All three now source the shared owner and take `require_python`, which is the right contract because every one of these call sites already exits on failure. The review cited one call in the notifier. There were FOUR — the control counts them rather than trusting the citation: pre-fix watch-tasks-stream.sh 1 · watcher_sentinel.sh 1 · task-notifier.sh 4 fixed 0 · 0 · 0 `tests/watcher-shells-resolved-interpreter.test.sh` adds the production control that was asked for: a good `SUTANDO_PY` with a FAILING `python3` first on PATH. Pre-fix that run executes the stub and prints its refusal; post-fix it resolves. The negative control holds too — with no runnable interpreter at all it refuses rather than guessing a path. pre-fix 4 of 5 fail, including the live one fixed 5 of 5 Not exposed on this host: `python-bin` and PATH `python3` both resolve to /opt/homebrew/bin/python3 here, so this is a defect for hosts where they differ, not a live outage. Said so rather than implying a fix to a running fault. Regression sweep: 47 suites touch these shells; 42 pass and the 5 failures (3 TypeScript, 2 sync-workspace) reproduce identically at the unmodified parent. Stand: Echo Act IV Pro * fix(health-check): `ps` can prove an identity absent, never prove a value Reported at the previous head. `ps` concatenates argv and env with spaces, so a value containing one is unrecoverable: `SUTANDO_INSTANCE_ID=worker 7` splits into `...=worker` and `7`, and the reader returned `worker`. Not a fails-safe error — the two are genuinely distinct keys, so the repair re-stamps a DIFFERENT instance's sentinel: instance_key("watcher-a", "worker 7") -> watcher-a+worker%207 instance_key("watcher-a", "worker") -> watcher-a+worker No parse fixes this; the ambiguity is in the format. So the fallback now answers only what it can prove: `""` when the environment was printed and the name is absent, and `None` — cannot read — whenever the name is PRESENT, since a recovered value may be a prefix of the real one. Measured on this Darwin host while confirming the report: `ps -Eww -p PID -o command=` exposes NO environment at all for a real owned shell child, even with both identity vars explicitly set. The re-stamp path is therefore already unavailable here, and this makes that unavailability honest rather than silent. Also delegates the actor-key precedence to the module that defines it. `rundir` exports `ACTOR_ENV_NAMES` and uses it in `agent_id`; health-check reads it via `util_paths.actor_env_names()` instead of carrying its own copy of `SUTANDO_AGENT_ID -> AGENT_MXID -> AGENT_ID`. The test asserts the two agree AND that no second spelling of the list survives in health-check. truncating reader restored 2 failures fixed 29 tests, OK (was 22) Two of the added cases are controls: a printed environment can still prove absence, and the two identities the truncation would have confused are distinct — without the second the finding would be moot. Stand: Echo Act IV Pro * fix(health-check): the watcher predicate is a shape, not a field count Reported at 6833f87f. `len(parts) != 2` rejected two shapes that actually run: the Codex notifier execs the script WITH a tasks directory, and an install path containing a space splits into more tokens again. A production watcher therefore read as "not a watcher" — so a duplicate is invisible to the tree count, and the re-stamp branch cannot see the process it is meant to repair. bash src/watch-tasks-stream.sh was True bash /repo/src/watch-tasks-stream.sh /w/tasks was FALSE bash /Users/x/Application Support/.../watch-tasks-stream.sh was FALSE It was also wrong in the OTHER direction, which the report did not name and the impostor control found: `endswith` accepted a DIFFERENT script. bash /repo/src/x-watch-tasks-stream.sh was TRUE The predicate now requires a shell, refuses `<shell> -c ...` (the self-match it has always existed to exclude), and matches the script as a whole final path component — so a similarly named neighbour and a `.bak` sibling both refuse. Nine cases, four real and five impostors; restoring the field count fails three of the four new tests. Every pre-existing case used the synthetic two-token form, which is why none of them could see either direction of this. field count restored 5 failures fixed 32 tests, OK (was 29) Stand: Echo Act IV Pro * fix(startup): reap this instance's sentinel, not every instance's Reported at 6833f87f. `reap_stale_task_watcher` deliberately KILLS a watcher that owns its sentinel — correct for a leftover of this instance, and startup looped it over every sentinel `sentinel_paths_in` returns. Reproduced with the production function and two legitimate owners: before A alive=yes B alive=yes (two sentinels, two live watchers) after A alive=no B alive=no both sentinels removed fixed A alive=no B alive=YES peer's sentinel intact The reaper rests on "startup runs once before any watcher", which stopped being true when a pool host got peers. Startup now resolves its OWN sentinel through `sentinel_path_for` and reaps only that. The existing wiring assertion pinned `sentinel_paths_in` — it encoded the defect, so it is replaced by its opposite plus a refusal if the enumeration returns. A LIMIT I would rather write down than let the count imply. The peer-survival case calls `sentinel_path_for` and the reaper DIRECTLY, so it proves the scoped shape is safe but does NOT re-fail if startup.sh regresses; restoring the loop failed exactly the two greps while that case still reported ok. The greps pin the wiring, the case pins the behaviour of what is wired. Exercising startup.sh's own block would need it extracted, which is a refactor and not this fix. enumeration restored FAILED (2), both wiring assertions fixed ALL PASS, 12 checks Stand: Echo Act IV Pro * fix(health-check): classify by the EXECUTED script, not any mention in argv qingyun-wu's probe at 7b66bc9d: `bash <tmp>/unrelated.sh /repo/src/watch-tasks-stream.sh` was classified True, so an unrelated process could be published as the watcher and the Stop-hook cleanup could terminate it. The predicate searched the whole flattened argv; it now tests the first `.sh` token, which is the script the shell executes. The space-in-path shape the docstring exists for still passes: `bash /My Path/src/ watch-tasks-stream.sh` reaches the watcher as its first `.sh` token. Control: restoring the previous line makes case aa fail with exactly that argv. Stand: Echo Act IV Pro * fix(watcher): read the executed script from the real argv vector, not the flattened string `ps -o args=` flattens argv, so a spaced install path and a script plus its arguments are the same string. The `.sh`-suffix heuristic that stood in for the missing structure got three of six real launch shapes wrong, including a false negative that made a running production watcher invisible to the tree count. Take argv as a vector where the OS offers one -- /proc/<pid>/cmdline on linux, KERN_PROCARGS2 on darwin -- and fall back to the string only when no pid is available. The fallback is now tri-state: a hit in the script token holds under every possible re-split and returns True, a hit only in a later token is genuinely undecidable and returns None. Callers take opposite defaults on None, which is why it is not a bool: the tree count treats UNKNOWN as a watcher (missing one starts a second and every task is then processed twice), while the sentinel repair declines (publishing a wrong pid gets a stranger killed). Stand: Echo Act IV Pro * fix(health-check): classify each watcher root by owner before advising a stop Parentage was computed and then used only when there was exactly one root, so every multi-root set was reported as orphaned with "stop them and restart one cleanly" -- advice that tells an operator to kill a watcher whose live parent is about to notice. Three branches gave it: the dead-sentinel branch, the untracked- extras branch, and the sentinel-less branch that already classified correctly for a single root two lines above. All three now split the roots they name into ownerless and supervised and advise a stop only for the first. One classifier does it, shared, rather than the per-branch copies that let the single-root case drift ahead of the others. The three tests that pinned the literal word "orphaned" now assert the split instead; a wording pin could not tell the two groups apart, which is how the advice stayed wrong under a green suite. Stand: Echo Act IV Pro * fix(health-check): a duplicate is a problem even when both roots are supervised The previous commit split multi-root advice into ownerless and supervised and told the operator to leave every supervised root alone. That is right for ONE root with a sentinel gap and wrong for two: two watchers means every task is processed twice, and a live parent does not make that acceptable. The change broke tests/health-check-supervised-watcher-not-orphan.test.py, whose two-tree case supplies two SUPERVISED roots and asserts the stop remedy survives -- the suite named for this exact behaviour, and the one I did not run. Multi-root keeps the duplicate verdict and the stop remedy. The ownership split now says which to stop FIRST and that stopping a supervised one needs its launcher, rather than implying a supervised duplicate may stay. Stand: Echo Act IV Pro * test(health-check): exercise the darwin argv parse on any platform KERN_PROCARGS2's layout is parsed by hand -- an exec path to skip, two runs of padding NULs, and an argc bound that stops argv from running into envp -- and on a linux runner /proc answers first, so none of it ever executed under CI. The coverage gate named lines 8380-8394 for exactly that reason. Five cases drive the branch by faking sysctl: the ordinary vector, padding between the exec path and argv, leading padding before the exec path, argc bounding the result so environment strings are not returned as arguments, and a failing sysctl returning None rather than a partial vector. `ctypes.util` is imported before CDLL is patched -- its module init calls CDLL and subscripts the result, so patching first makes the import itself fail. Stand: Echo Act IV Pro * test(health-check): cover the identity probes' cannot-read and states-none paths The coverage gate named the remaining lines and every one is a tri-state edge: `_pid_env_first` returning None when neither /proc nor ps can answer versus "" when the process states no value, `_watcher_sentinel_target` refusing when no default identity is stated or the resolver raises, and `_ps_watcher_index` skipping a row with no argv column. They matter more than their line count: collapsing None into "" is how a repair publishes THIS process's identity as some other pid's sentinel, and that pid is what the Stop hook kills. `actor_env_names` and `stated_default_identity` get the same treatment in util_paths -- with the runtime module absent they must state absence rather than fall back to the caller. The util_paths file carries a third case asserting both return something when the module DOES resolve, because two null-path tests would otherwise pass on a broken import. Stand: Echo Act IV Pro * fix(availability): the session boundary reads the writer's path, not a literal The sentinel writer names its file per instance; `_session_started_at` still stat()'d "watch-tasks-stream.pid". A restarted core then read session_start=None and scored a previous session's RUNNING snapshot as current work, reporting busy_unavailable until that snapshot's freshness expired. Routed through `watcher_sentinel_path`, the same owner the writer uses. It returns the bare historic name when the scope key is empty, so an unmigrated host is unchanged. NOT fixed by selecting the newest peer sentinel, per the report — a test pins that a sibling's file is never adopted. M1 revert to the historic literal FAILED (failures=1) M2 the forbidden newest-peer-sentinel FAILED (failures=1, errors=9) baseline / restored OK (35 tests) 45 suites import the touched modules; all pass. runtime-api-server-branches fails identically at the pristine head and references neither module. Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(watcher): compare the exact final component, not the name anywhere in it The vector branch matched the script name anywhere in vec[1] through a regex whose `[\s/]` alternative treats a space as a component boundary. Inside one authoritative pathname it is not: `backup/copy watch-tasks-stream.sh` is a file named `copy watch-tasks-stream.sh`, and `backup/watch-tasks-stream.sh backup` is a file named `watch-tasks-stream.sh backup`. Both were recognised and both published a pid, pointing sentinel repair at an unrelated process. Reporter's table, all five, now hold on the vector path: plain/src/watch-tasks-stream.sh True control spaced dir/src/watch-tasks-stream.sh True control — a spaced DIRECTORY still counts backup/watch-tasks-stream.sh backup False was True, published a wrong pid backup/copy watch-tasks-stream.sh False was True, published a wrong pid backup/watch-tasks-stream.sh.bak False control M1 restore the regex FAILED (failures=2) M2 substring not basename FAILED (failures=2) baseline / restored OK The string fallback is UNCHANGED and still returns None where a spaced pathname and a script-plus-arguments are the same bytes. Tightening it there turns the ordinary production launch into None and a pinned test says so; that trade needs the reporter. Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(watcher): an unreadable argv vector says UNKNOWN, never authorizes repair With the OS argv vector the executed pathname is exact. Without it a flattened argv cannot tell `bash /repo/src/watch-tasks-stream.sh /w/tasks` from `bash '/repo/watch-tasks-stream.sh backup'` — the same bytes. The fallback returned True for both, so a vector-read failure published an unrelated pid. True now only when argv ends at parts[1]; otherwise None. OS argv available recognized=False same, vector read=None recognized=None (was True, published a wrong pid) UNKNOWN published=False _watcher_trees(UNKNOWN) {'4242': {'4242'}} -- counted, not dropped The directed relaxation of test_every_real_launch_shape_is_recognised to `is not False` is satisfied by True, so on its own it left the fallback UNPINNED — M1 passed against it. Added AnUnreadableVectorSaysUnknownNeverTrue asserting the positive shape instead. M1 restore unconditional True FAILED (failures=1) M2 reject the 2-token shape FAILED (failures=2) baseline / restored OK Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(watcher): never tell an operator to stop a supervised watcher The multiple-root warning called every root "orphaned", said "stop them and restart one cleanly", then appended the ownership split without retracting that instruction. With two supervised roots it read as "stop both healthy instances". Advice is now derived from ownership: ownerless roots are named as stoppable; with none, it says do NOT stop any and points at the owning launcher. The reason a duplicate matters is kept — the detail still states every task is processed Nx. Reporter's fixture (901/902 supervised by 900): before "2 orphaned watcher(s) ... stop them and restart one cleanly. ownerless, stop these first: none; supervised, ...: 901, 902" after "2 watcher(s) ... every task is processed 2x. do NOT stop any of them — each is supervised; reduce the count through the launcher that owns it. ownerless: none; supervised: 901, 902" M1 restore the blanket stop advice FAILED (failures=1) M2 never name an ownerless root FAILED (failures=1) baseline / restored OK M2 is the control against over-correcting into "never stop anything": a genuinely ownerless root is still named. Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(watcher): keep the orphan verdict for ownerless roots, drop it for supervised 09363932 over-corrected. It removed "orphaned" and "stop them" from EVERY multi-root verdict, including genuinely reparented watchers (ppid 1) and unknown- parent ones, where both words are true and the stop remedy is right. CI caught it: health-check-supervised-watcher-not-orphan failed 5 assertions. The vocabulary now follows ownership, not just the advice: ownerless present -> "N orphaned watcher(s) ... stop them and restart one cleanly" all supervised -> "N watcher(s) ... processed Nx. Do NOT stop any of them: each is supervised; reduce the count through its launcher" Two assertions in that suite pinned the DEFECT and are inverted, with the reason inline: v3 is two roots with LIVE parents (99, 98), so "2 orphaned" is false and "stop them" tells an operator to take two healthy instances offline. The duplicate is still asserted — "2 watcher(s)" and "processed 2x" replace it. supervised-watcher-not-orphan 5 failures -> PASS pool-watchers-all-tracked PASS agent-availability PASS 69 of 72 suites touching check_task_watcher/_watcher_trees/orphan: all rc=0 (3 unrun — the sweep hit a 2-minute cap, not a failure) Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(watcher): scope stop advice to the ownerless subset, not the whole root set e20458ce fixed the all-supervised case and left the MIXED case carrying the same harmful advice: the `if ownerless:` branch formatted len(roots) and every PID, so with 901 supervised (ppid 900 live) and 903 ownerless (ppid 1) it said "2 orphaned watcher(s) (pids 901, 903) ... stop them" -- telling an operator to take healthy peer 901 offline. Three ownership cases now, not two: mixed -> "Stop ONLY the ownerless (903) ... Do NOT stop 901 - supervised" all ownerless -> "N orphaned watcher(s) (pids ...) ... stop them and restart one cleanly" all supervised -> "Do NOT stop any of them: each is supervised" The count and the per-task duplicate cost are stated in every case; only the remedy is scoped. test_an_ownerless_root_is_still_named_as_stoppable was pinning the blanket instruction for the mixed case, so 55 green tests did not protect PID 901. It is replaced by two: the mixed case (903 stoppable, 901 explicitly preserved) and an all-ownerless control so the fix cannot become "never stop anything". pool-watchers-all-tracked 56 tests OK supervised-watcher-not-orphan PASS perturbation control: reverted to the blanket form -> the mixed-case assertion FAILS (1 failure), restored -> OK. The assertion is exercised, not assumed. Reported-by: qingyun-wu Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * fix(health-check): identity decides a duplicate, and one predicate decides a watcher keweichen's two P1 findings at ae7e4d2f, both of which let an operator act on a fact the check never resolved. 1. A root count is not an identity. The untracked-tree branch turned two roots into "duplicates process each task more than once" and named stop targets, without consulting `_watcher_sentinel_target`. For two valid instances the advice has opposite failure modes: same instance -> restarting recreates the duplicate; distinct instances -> reducing removes the only watcher for one. Classify each extra root by its resolved sentinel target — same target is a duplicate, a different target is a separate instance missing its record, and an unresolvable one renders UNKNOWN with no stop or reduce instruction. 2. Sentinel validation trusted `"watch-tasks-stream" in argv`, which the module's own `_is_watcher_argv` rejects. A stranger carrying the script name as a DATA argument was counted live, so a missing watcher read healthy. Use the authoritative predicate, and honour its third state: None is neither live nor PID-reuse, and authorises nothing. Controls, each mutant scoped to the branch it targets: drop the UNKNOWN branch FAILED (failures=2) treat a distinct target as a duplicate FAILED (failures=1) restore the argv substring test FAILED (failures=1) baseline OK (59 tests) The pre-existing untracked-root case supplied no identities and so pinned the count-implies-duplicate premise; it now carries a resolved same-target identity, and the distinct and unreadable schedules are pinned beside it. Stand: Echo Act IV Pro * fix(health-check): the no-sentinel branch classifies identity too qingyun-wu, #3875: identity-based classification repaired the tracked-sentinel branch, but the no-sentinel branch above it still derived "every task is processed 2x" and "reduce the count" from the number of roots alone. Losing the last sentinel therefore turned two distinct instances back into duplicates, and following that advice can remove the only watcher for one of them. Both branches now classify through one helper, `_group_roots_by_target`, rather than a second copy of the policy: duplicate N roots share one resolved target -> states the Nx cost distinct every root resolves to its own -> "not duplicates", no reduction unreadable any root has no (agent, instance) -> UNKNOWN, no reduction Ownership-scoped stop advice is unchanged: an orphaned watcher is still reportable on ownership grounds. Only the DUPLICATE claim and the affirmative reduce instruction are gated on identity, which is what the finding is about. Controls: no-sentinel branch ignores identity FAILED (failures=2) DISTINCT verdict removed FAILED (failures=1) reduce instruction returns unconditionally FAILED (failures=2) baseline OK (61 tests) The pre-existing duplicate-cost case asserted "processed 2x" for two roots with no identity at all — the premise being fixed. It now supplies a shared target, so the claim it makes is true, and the distinct and unreadable schedules are pinned beside it. Stand: Echo Act IV Pro * test(health-check): the supervised-roots suite states identity too The duplicate-cost check asserted "processed 2x" for two roots with no resolvable (agent, instance) — the same count-only premise the production fix removes, in a second suite. Give it a shared target so the claim it makes is true, and pin the distinct-instance verdict beside it. Stand: Echo Act IV Pro * fix(health-check): reduction reads the supervised subset, in every ownership branch qingyun-wu, #3875: the mixed-ownership branch selected its advice before the identity guard ran, so with distinct supervised instances it said both "do not reduce the count" and "reduce those through the launcher" in one report. Reduction is advice about the SUPERVISED subset, so that subset's identity is what licenses it — never the whole root count. Both the mixed and all-supervised branches now share one test, and the orphan reporting they wrap is untouched. mixed, supervised distinct Stop ONLY the ownerless; no reduce offer mixed, identity unreadable UNKNOWN; no reduce offer mixed, supervised share a target processed 2x; reduce those (advice survives where true) Controls: mixed branch reduces unconditionally FAILED (failures=4) subset test widened to every root FAILED (failures=1) baseline OK (65 tests) Two test defects of my own, both reported by the same review. The new supervised-suite checks ran AFTER `if failures: sys.exit(1)`, so forcing one red printed FAIL and still exited 0 — moved above the gate, and a forced failure now exits 1. And both harnesses left the production identity resolver in place when a fixture named no targets, which makes the verdict depend on the host: green here, red in CI at the same commit. `targets={}` now states "no pid resolves"; only the restamp cases, which mean to exercise the real resolver, pass None. Stand: Echo Act IV Pro * fix(health-check): ownerless stop advice needs identity, and so does restart count keweichen at e3fbc2cf: parentage establishes supervision; it establishes neither target identity nor restart cardinality. Three ways the ownerless branches acted without either: two DISTINCT all-ownerless instances "stop them and restart one" strands one an UNKNOWN ownerless root named safe to stop ownerless sharing a SUPERVISED target "stop it and restart one" recreates the duplicate the stop just removed The ownerless subset is now grouped by resolved target, exactly as the supervised subset already was: unresolvable Do NOT stop — it may be the only watcher for its instance N distinct targets restart N cleanly, one per instance, NOT one target already served stop, and do NOT restart one instance, one root stop and restart one (the remedy survives where true) Controls: ownerless advice ignores identity FAILED (failures=4) restart cardinality pinned to one FAILED (failures=1) restart even when a supervised root serves FAILED (failures=1) baseline OK (69 tests) Four pre-existing cases asserted the old advice for roots with no identity at all — the same premise, a third time. Each now states its identity, and the unidentified variants assert the refusal instead. Stand: Echo Act IV Pro * fix(health-check): sentinel reconciliation keeps every record, including UNKNOWN keweichen at e3fbc2cf, three ways a sentinel record was lost or a fault silently dropped: two sentinel files naming one pid `live[spid] = sp` kept the last and reported `ok: streaming watcher alive` for both an unprovable sentinel beside a `unprovable` was absent from `faults`, so a live one live peer produced a false green a dead sentinel plus ownerless the remaining branch still named stop targets watchers without resolving their identity Collisions are recorded rather than overwritten, `unprovable` joins the fault list, and the dead-sentinel branch drops unresolvable roots from "safe to stop" and names them as UNKNOWN instead. A live peer never clears another instance's record — that was already the stated contract here; two of these three broke it. Controls: collision tracking removed FAILED (failures=1) unprovable dropped from faults FAILED (failures=1) baseline OK (71 tests) The unprovable control needed a hook: no argv string reliably yields the tri-state None, so `_is_watcher_argv` returned False and the case exercised the `reused` path instead. Without forcing the verdict the test passed against a mutant that removed the very branch it named. Stand: Echo Act IV Pro * fix(services-status): two sentinels naming one pid is not two watchers john-the-dev on #3875: the PID-keyed collapse had a second reader. While check_task_watcher dropped the duplicate record, services_status reported the pool that does not exist: before ('running', '2 watchers: pid 95442, pid 95442') after ('degraded', 'one pid, two sentinels: …+a.pid and …+b.pid both name pid 95442') Fixing only the first reader would have left the two disagreeing about the same sentinel directory, which is how they drifted apart to begin with. Control: with services_status reverted, the new case fails on that exact fixture; restored, the suite passes. Stand: Echo Act IV Pro * fix(health-check): ownerless remediation is per target, from one identity snapshot keweichen at 6f460441: the branch handled target sets all-or-nothing and missed two adjacent inputs. partial overlap supervised 901->A, ownerless 903->A + 904->B said "restart 2", recreating A beside the live 901. Only the UNCOVERED targets need a restart: covered ones are already served and must not be. UNKNOWN on the supervised 901->UNKNOWN, ownerless 903->A said "stop and SUPERVISED side restart one". If 901 is A the restart duplicates it; if it is not, stopping 903 leaves an A gap. Neither is decidable, so no action is licensed. Both now derive from ONE identity snapshot taken at the top, rather than three separate resolver reads that could disagree if a process's identity changed between probes — which was the same review's third point. Controls: partial-overlap branch removed FAILED (failures=1) unknown-supervised gate removed FAILED (failures=1) baseline OK (73 tests) Stand: Echo Act IV Pro * fix(health-check): a sentinel fault vetoes remediation, and is not skipped by it keweichen at 70a8887d: the extras schedules returned before the shared fault aggregation, so an unprovable or conflicting sentinel record never reached the report when an untracked tree was also present. With a live sentinel A->100, an unprovable B->200, and BOTH trees visible — which production retains, since an argv-UNKNOWN tree is not discarded — the check said the process "belongs to a DIFFERENT instance" whose sentinel is missing, though B's sentinel exists. Where both resolved to A it went further and called pid 200 "ownerless, safe to stop". Faults are now built before the extras schedules, and an unprovable or colliding record vetoes stop and restart advice about untracked trees. The prior reconciliation test omitted the unprovable pid from `trees`, so it never entered the branch that defeated the fix — the fixture avoided the interaction it was written to cover. It now supplies both trees. Controls: veto removed FAILED (failures=1) faults empty at the veto point FAILED (failures=1) baseline OK (74 tests) Stand: Echo Act IV Pro * fix(health-check,services-status): name which watcher to restart, and never strand one keweichen at 70a8887d, three findings outside the two rows already repaired. 1. Partial overlap computed the right uncovered target and then discarded its identity: "restart 1 cleanly" does not say restart 904/B rather than 903/A, so an operator can recreate A beside the live 901 and leave B absent. The instruction now names both dispositions per pid. The prior test asserted only cardinality. 2. With every sentinel dead, an ownerless root is its instance's ONLY watcher unless a supervised peer serves the same target — "safe to stop" stranded it while its sentinel was already down. Those roots are excluded and named. 3. services_status detected the two-sentinel/one-pid collision only when every row was running. A/B -> live 100 with C -> dead 999 reported "2 of 3 up (pid 100, pid 100)": the collision vanished and one process was counted twice. Now "1 of 3 up (pid 100); …; one pid, two sentinels: …". Controls: per-pid disposition reverted to a count pool FAILED sole-watcher exclusion removed pool FAILED services collision note dropped services FAILED baseline both OK (75 tests + services suite) Stand: Echo Act IV Pro * fix(health-check): one restart per uncovered target, and extras keep their multiplicity keweichen at 3d9d11e2, two more the previous fixtures could not distinguish. 1. The restart mapping selected EVERY pid on an uncovered target. With 903->A, 904->B and 905->B it said "restart 904, 905", recreating B's duplicate. One representative per target now, and the reasons are separated: a root skipped because a supervised watcher serves it is not the same as one skipped because a peer above already covers that instance. The prior test had a single root on B, so it could not tell "one per target" from "every root on a target". 2. Extras distinct from every TRACKED target can still duplicate EACH OTHER. Tracked 100->A with extras 200->B and 300->B read "not duplicates; each is missing its own sentinel record" — erasing that 200 and 300 are one instance twice. They are now named as sharing an instance, with "keep ONE of each and stop the rest"; the "Do NOT stop them" advice is kept for the case where the extras really are one apiece, since the two instructions contradict. Controls: restart every root on an uncovered target FAILED (failures=1) extras multiplicity erased FAILED (failures=1) baseline OK (77 tests) Stand: Echo Act IV Pro * test(pool): assert multiplicity survives to the diagnosis, as a matrix keweichen's generalisation of every finding on #3875: each was one shape — a multi-map reduced to a set before the multiplicity decision was finished. A target set losing its roots, extras compared only to tracked targets, an early return dropping classified records, a collision return dropping a live pid. So state the property once over a matrix instead of adding a regression row per report: wherever N roots share a target the verdict must still be able to say N, and must never advise restarting more watchers than there are distinct uncovered targets. Control: flattening `target -> [roots]` back to a set turns it red. Stand: Echo Act IV Pro * fix(health-check): every classified sentinel record survives the extras path keweichen at 3d9d11e2: the veto added at 99b4c8af composed the fault list with the extras schedules only for `unprovable or collided`. A dead, reused or unreadable record still vanished through the same returns — the narrower fix repaired the instance reported rather than the shape. Any classified fault now reaches the report and holds off advice about untracked trees, with the reason distinguishing an unknown record from a known-bad one. Control: narrowing the veto back to unprovable/collided turns the new dead-record case red. Stand: Echo Act IV Pro * fix(health-check): peer-duplicate advice respects supervision qingyun-wu at 5425c7ad: the peer-duplicate advice added for extras — "keep ONE and stop the rest" — chose its survivor by position and never consulted ownership, so the report could prescribe stopping a watcher with a live parent. That is the defect this PR exists to prevent, reintroduced by my own fix one commit earlier. Ownership now decides which peer stays: mixed keep the supervised, stop the ownerless all supervised reduce through the launcher that owns them; do NOT stop them all ownerless keep ONE and stop the rest (unchanged) Control: dropping the ownership split turns both new cases red. Stand: Echo Act IV Pro * fix(health-check): aggregate every sentinel record before any no-live advice keweichen's P1-1 at a5933d4f: each no-live early return sat before the `faults` aggregate, so the classes below it were discarded. With an unreadable sentinel AND an unprovable one, the first branch returned "restart the watcher" and the UNKNOWN vanished -- licensing a restart that can duplicate the very watcher whose argv could not be read. Reused+unprovable and dead+other-fault lost records the same way. Every class is now collected before any advice, and an unprovable identity vetoes BOTH directions: no restart (it may duplicate the live watcher) and no stop list (it may kill it). The roots branch carries the same veto. Controls, his schedule: unreadable only restart-advice YES (unchanged positive control) unprovable only restart-advice no (unchanged positive control) both together YES -> no, and BOTH records now survive Five cross-product tests added; four fail against this parent, the fifth is the single-fault positive control that must pass either way. 81 -> 86 tests, and health-check-task-watcher / supervised-watcher-not-orphan still exit 0. P1-2 (tracked-target duplicate hiding peer duplicates) and P1-3 (enrolled default-instance key) are not in this commit; all three ship in one push so a standing approval is spent once rather than three times. Stand: Echo Act IV Pro * fix(health-check): partition every target group, so one duplicate cannot hide another keweichen's P1-2 at a5933d4f: peer multiplicity was analysed only under `if distinct and not dupes`, so a duplicate on a TRACKED target bypassed the branch entirely and a duplicate on a PEER target disappeared into the words "a different instance, left alone" -- leaving that peer processing every task twice. His schedule, sentinel A on 100, extras 200->A, 300->B, 400->B: parent A-tracked + A-dupe + B + B B pair NOT named HEAD A-tracked + A-dupe + B + B B pair named, with ownership both peer-only B + B (control) named -- unchanged The computation moves into `_peer_multiplicity_parts()` and runs on BOTH paths rather than one, so the two questions (does anything duplicate a tracked watcher / does anything duplicate a peer) are answered independently. Four tests in their own class, three failing against this parent; the fourth is the peer-only control that must pass either way. 86 -> 90. task-watcher and supervised-watcher-not-orphan still exit 0. P1-3 (enrolled actor in the sentinel key) still open; all three ship in one push. Stand: Echo Act IV Pro * fix(util-paths): the canonical actor is the enrolled one, so an upgrade renames nothing keweichen's P1-3 at a5933d4f: the no-migration compatibility claim was false for an enrolled default-instance install -- the ordinary single-host shape. The key compared the resolved actor against the bare DEFAULT_ACTOR constant, so any host with auth/ag2space.json renamed BOTH durable records on upgrade: empty state dir watch-tasks-stream.pid -> unchanged enrolled watch-tasks-stream-@enrolled:ag2.space.pid <- renamed fallback receipts task-event-handler-fallbacks/@enrolled:... <- renamed The historic sentinel then stays enumerated as a dead peer, and durable receipts change namespace. Taken his second remedy -- make the historic-name claim TRUE rather than write a migration. `stated_default_identity()` ALREADY defines the right notion: the canonical actor for a state dir is its enrolled agent where one exists. The key compared against the constant instead, so the two disagreed. Aligning them fixes both records at once and needs no transition. It does not cost what the change is for: on an enrolled host, worker-1, worker-2 and default still produce three distinct sentinels. Three tests on an ENROLLED fixture -- the shape the existing compatibility test misses, since it uses an unenrolled empty state dir. Two fail against this parent; the third (instance separation) is the control that must pass either way. Naming 27 -> 30. 46 suites importing util_paths / instance_scope_key / watcher_sentinel_path / handler_fallbacks_dir: 45 pass; runtime-api-server-branches fails identically on this parent (host-dependent, pre-existing, not this change). Stand: Echo Act IV Pro * style: trim three comment blocks to the repo's 2-line cap review-checks refused the push: the constraint belongs in the code, the narrative in the commit body. No behaviour change; both suites still pass. Stand: Echo Act IV Pro * fix(watcher): STATE_DIR follows WORKSPACE_DIR, so a scratch run cannot eat the checkout sentinel keweichen's P1 at c40e6e61: the fallback-receipt integration test wrote and then deleted a real installation's watcher sentinel. Reproduced exactly -- pre-seed the canonical sentinel with 777, run the test, and it is gone: head c40e6e61 before=777 after=MISSING this commit before=777 after=777 Root cause is in production, not the test. `WORKSPACE_DIR` is derived from the argv tasks dir at :82, and RESULTS_DIR, CLAIMS_DIR and FALLBACKS_DIR all follow it. STATE_DIR alone re-resolved the CHECKOUT workspace, so an argv tasks dir moved every other state path and left this one pointing at the live install -- while the comment above it claimed the resolutions were the same. Took keweichen's second option: make the comment true. No production behaviour change: with no argv, TASKS_DIR is the resolved workspace's tasks dir, so WORKSPACE_DIR is that workspace and STATE_DIR is byte-identical. Only an argv override moves it, which is the test case. Control added and validated both ways: a pre-existing checkout sentinel must be byte-for-byte untouched. It FAILS against this parent (after=None) and passes here; 4/4 -> 5/5. The concurrency symptom he measured is also gone -- the pool suite run CONCURRENTLY with this test is 90/90, where it was 18/90 failing. Stand: Echo Act IV Pro * style: trim a comment block to the repo's 2-line cap review-checks refused the push; no behaviour change, control still 5/5. Stand: Echo Act IV Pro * test(watcher): the receipt-scope control is read-only on the resolved workspace qingyun-wu at 74037add: the control seeded the canonical sentinel and then unlinked it, so a watcher starting mid-test lost its sentinel (proved with a line-trace hook: before=b'sentinel-control-do-not-touch\n' after=b'3462\n', then watcher_alive=True sentinel_exists=False). The canonical path is now only ever read -- exists() and read_bytes(), before and after -- and every write in the file targets the scratch workspace. The scope assertion moved to the scratch state dir, sampled WHILE the watcher is alive, because the watcher's cleanup trap unlinks its own sentinel on exit and a post-hoc listing is therefore always empty. Discriminates: 5/5 at HEAD, and the scope check FAILS against c40e6e61's watcher source ("scratch state held []"), so it still fails for the original reason. Stand: Echo Act IV Pro --------- Co-authored-by: Chi Wang <sonichi@users.noreply.github.com> Co-authored-by: Qingyun Wu <qingyun0327@gmail.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ehind an adapter, one transactional task store The normative v1 worker-pool design, restructured around the owner's 2026-09-07 review of the #3860 draft: a single Sutando pool supervisor that depends on no LLM session is the only scheduler; core and workers are executors behind Executor.offer/accept/start/cancel/events/health with Claude, Codex App Server and ACP adapters; task control state is one SQLite table with PENDING -> OFFERED -> ACCEPTED -> RUNNING -> SUCCEEDED | FAILED transitions as single conditional updates; explicit accept plus lease, supervisor-owned reconciliation of expired leases; room-to-worker binding with requested_worker override; bound-but- unavailable rooms stay PENDING with Rebind / Restart / Process with core surfaced to the owner; a four-state health model with a probing transition on kick. The record behind it — rejected alternatives, the failure model taken from the #3860 draft, and the owner decisions — lives in a separate notes file, so the normative text carries no review history. Tests: a retraction suite that rejects the superseded mechanisms (lead or router in the daemon, core as scheduler, watcher-owned admission, filename-keyed claims, per-worker proactive loop, the file-protocol directories, TASK_FILE as the delivery abstraction) each with a live positive control, and a transitions model of the store's state machine with a lease clock, crash seams, supervisor restart and the four crash windows.
…expiry; carry remaining #3860 coverage Review (pullrequestreview-5138901924): - Row 6 now writes the result and the outbox record (delivery intent, keyed on the task id) before clearing the lease identity, so a crash in the seam re-drives from the outbox and never drops a completed reply. Recovery re-drives outbox records without a delivered sentinel. - Expiry is state-specific: an OFFERED record expires on offer_expires_at, an ACCEPTED/RUNNING one on lease_until. Fixes the reconciliation step and the non-terminal invariant that named only lease_until, which would strand an offer no executor accepted. The transitions model now carries offer_expires_at as a distinct field and adds two crash-seam tests (offer expiry; a completed reply surviving the identity clear). - The terminal owner-decision note is reframed as the design's own rationale rather than an unverifiable quotation of a private discussion. #3860 coverage carried so #4037 fully supersedes it: - the per-task timing log data/pool-metrics.jsonl as the saturation substrate; - auth (401 -> recycle) and transport (5xx -> back off) worker-failure handling; - the pin lookup keying on the provider-native room id (channel_id / chat_id). Guard tests: retractions 14, transitions 22, docs-audit 5. No model or rule regressions; the journal remains the store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
The owner asked for #3860's requirements at the top of the design doc in exact words. Reproduced verbatim, ahead of the design, with the source PR cited. Two conflicts with the design below are flagged and left UNRECONCILED for the owner rather than merged silently: - the requirements name a router installed with worker 1; the design names a pool supervisor that is the only scheduler, and #3860's own title says 'no router process' - the requirements are dated owner-decided 2026-09-03, the design 2026-09-07 Nothing in the existing design was altered; the file is prepend-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
The owner asked that conflicting inputs be marked in a discussion section
rather than left as an inline aside.
Four substantive conflicts, each with both sides quoted:
1. task files renamed (requirements) vs immutable (design) — the two
on-disk protocols are not implementable together
2. atomic rename + done-flags vs lease + generation as the at-most-once
primitive
3. whether a scheduler exists before the first worker
4. whether workers keep draining while the scheduler is down
Naming (router vs pool supervisor) follows from those, so it is not
settled separately. What is NOT in dispute is listed too, so the open
question stays as small as it actually is.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ne record Two requirements the #3860 coverage check found uncovered. Retirement touches two records -- roster and folder -- and no two file writes are atomic together, so it is ordered and every step idempotent: roster first (the only decision), then a per-tick sweep moving the folder's .txt back to tasks/ and leaving .claimed to finish or release, then rmdir only when empty. Nothing under deliveries/ is deleted unread. A router pass racing the retire strands nothing: the next tick sweeps it. The partial states reproduced against the rescue line cannot arise: there is no token, no gate directory, no second record -- the folder's contents are the state. A table of every question with its one authoritative record and what it must never be inferred from. A second answering record is a defect while the two still agree; three review rounds on the rescue line found that class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com>
…r once (#4041) * docs(pool): the four contracts the design states three times but never once Each section here is a rule #3860 already enforces in several places without naming, plus one gap that composing three of its own rules exposes. - One authoritative source per fact: a table of question -> sole source over the artifacts the coordination contract already names. The rule was doing work in three places; stating it once is cheaper than deriving it a fourth time. Where v1 knowingly breaks it (the hard-link claim AND the .claimed-<name> rename) the section says so, with the reason and the cost, rather than claiming an exemption. - Executor reports: there is no process to report to, so the order of an executor's durable writes IS the protocol. Names what a lease generation would buy that never-clobber does not -- it refuses a second writer, not a late one. - Executor interface and runtime adapters: the boundary a second runtime needs, written against v1's real delivery paths rather than an offer/cancel API it does not have. - Delivery is not admission: the four-row table's `claim live, no accept -> leave it` is the only row with no exit, so an offer nobody took is indistinguishable from work in progress for as long as the publishing watcher survives. Stated as an obligation on the implementing PR, not solved here. Not ported from the parallel draft: the claim/accept deadlock (this document already carries the normative version, and a second copy is the defect the first section names) and the file-protocol-without-transactions rejection (it exists there to reject THIS design; its transferable half is already in the ordering rules). Stand: Echo Act IV Pro * docs(pool): the done flag fences a repeat, it does not report a completion The source-of-truth table asked "has an external side effect already happened" of the done flag. The flag is written BEFORE the effect -- :1341 ("durable before any external side effect; the at-most-once floor is unchanged"), :1733 ("it is about to cause an external effect"), :1740 ("durable before any external side effect, so a reclaim behind it cannot repeat one") -- so it cannot answer that question. The schedule that falsifies the old row: create done flag -> crash -> external effect never runs observed: flag=yes, effect=no The row now asks what the flag does answer, and answers it correctly under that same schedule: flag present -> a reclaim may not repeat the effect. That is the at-most-once floor the parent contract already states, so the table stops disagreeing with the section around it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> Stand: Echo Act IV Pro * docs(pool): name the flag-before-effect seam the fence leaves open The row rename said what the done flag proves. It did not say what stays UNKNOWN across the seam the ordering opens, which was the second half of the request. Durable-before-the-effect makes `flag=yes, effect=no` a reachable crash shape. The table reports no row for the effect's outcome, and the acknowledgement that the work completed is the result file, written afterwards -- so a reader treating the flag as a completion receipt turns an unperformed action into a reported one while the fence suppresses its retry. Where an outcome must be known, that is a separate acknowledgement with its own crash semantics, not this flag. Stand: Echo Act IV Pro * docs(pool): a revoked late writer's result is selected, not harmless keweichen's REQUEST_CHANGES on 5334645 is correct and is fixed here rather than argued. The doc claimed the duplicate result from a revoked late writer "is made harmless" because the done flag stops the external effect happening twice. That conflates two things. The flag fences the EFFECT; it does not select which body is authoritative. `find_result` in src/local_task_protocol.py reads the live path first and falls back to the archive only when it is absent -- its own docstring says archival trails delivery -- so a revoked writer's `os.replace` onto the same path is returned in preference to the winner's, before delivery and again on any later retrieval. Verified against the code rather than adopted: find_result is live-first, and the design's own publication recipe is temp-file + os.replace onto that same path. So v1 now NAMES the open obligation instead of asserting it away: the implementing PR owes a durable selector (generation, epoch or claim token checked at publication) and must pin both late-writer orders. A first-writer-only primitive is explicitly called insufficient, since it refuses a second writer while admitting a late revoked one. Three retraction pins added. Perturbation-verified: restoring the old "it is made harmless" sentence fails the suite (102 tests, 1 failure); with the fix, 102 OK. Transitions suite 71 OK, unchanged. Stand: Echo Act IV Pro * docs(pool): the never-clobber invariant is scoped, and result publication is its exception The paragraph enumerates three publications -- claim, accept record, probation allowance -- and then asserted the rule over "every publication". Result publication is temp-file plus a clobbering os.replace, and the section 90 lines below says a revoked late result is not refused, so both normative claims could not hold. An implementer reading the universal form would omit the durable selector the later section says is still owed. Scope the claim to the three it enumerates and name result publication as the explicit exception until the selector lands. The two new pins fail on the previous revision (107 tests, 2 failures) and pass here. The existing retraction pins could not: they assert only the retraction and open-obligation phrases, so they went green with the contradiction standing -- which is why the contradiction survived a green suite. Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): delete the truncated duplicate admission paragraph qingyun-wu, inline on docs/worker-pool-design.md (3965049983, [P3]): the parent merge preserved the complete paragraph and left a second copy that ends mid-sentence at "vanishes from the next". Confirmed before deleting: the complete text is at 552-559 and the fragment at 595-598 is byte-truncated, not a variant. Only the fragment is removed, so the delivery section now flows into the claim-record paragraph as asked. "vanishes from the next" occurrences: 2 -> 1. Both suites still pass at this head (retractions, transitions). Stand: Echo Act IV Pro * docs(pool): the gate must carry the obligations this layer adds, not only the parent's keweichen at 821205f: the base sync brought the parent's stage gate into this child, and that gate enumerates only the PARENT's obligations. This layer adds two more, so the gate could have lifted step 2 with both still open. His scan of the merged document: gate-is-a-read: document=2 stage_gate=2 expiry on the untaken state: document=1 stage_gate=0 durable selector: document=1 stage_gate=0 must pin BOTH late-writer: document=1 stage_gate=0 Both are now in the table and in a paragraph naming them as this layer's: - untaken-offer expiry: `claim live, no accept -> leave it` has no exit while the publishing watcher LIVES, and no v1 mechanism ends that state. Blocks steps 2 and 3 — the handler consumes offers, the sweep recovers them. - durable late-result selector: result publication is the stated exception to one-caller-wins, a clobbering os.replace does not refuse a revoked late writer, and both late-writer orders are owed a pin. Blocks step 2. Also merges the parent forward, bringing the fifth obligation (retirement crash-completeness) and the exhibited-evidence rule this child's gate lacked. His scan re-run against the fixed gate, over the gate's true extent rather than a fixed-size window: 2/2, 3/4, 2/3, 1/1 — every requirement now named inside it. Both suites pass. Stand: Echo Act IV Pro * test(pool): pin the gate mappings this layer adds, so removing them fails keweichen at 740b171: the gate repair had no discriminating pin. Removing the mappings and the child-obligations paragraph recreated the exact omission and both suites stayed green — 110/110 and 82/82. A repair whose absence is undetectable is the condition the gate itself was written to forbid. Four gate-SCOPED assertions, reading the blockquote rather than the document, because the defect was prose-present and gate-silent: - untaken-offer expiry appears in the step 2 AND step 3 rows - the durable late-result selector appears in the step 2 row - the gate requires BOTH late-writer orders - the gate names these as added by this layer, so a future parent merge cannot silently drop them the way the first merge did Control, his exact removal: 4 FAILURES. Restored: 114/114 and 82/82, with 114 defined == 114 collected. Also his two non-blocking items: the stray `>` fused onto the paragraph end (it had swallowed the blank quote line before "Step 5's"), and the inherited "four items below" at :29, now count-free for the same reason the heading is — a counted phrase re-stales every time the set changes. Stand: Echo Act IV Pro * test(pool): assert the normative phrase, not a substring it also matches keweichen at 1c135a5: `assertIn("late-writer order")` also matches "one late-writer order", so his one-word mutant passed 114/114. The test was named `requires_BOTH` and did not require BOTH. Full-paragraph removal — the control I did run — is not the adjacent alternative this requirement distinguishes. Now asserts the exact `BOTH late-writer orders` and refuses the singular form. His mutant: 1 FAILURE. Restored: 114/114 and 82/82. Also the stale count he quoted, which I had told him was not there. `:633` reads "The last-worker removal order below is one of the four" — the DISPUTED marker and the count share one long line, and my grep enumerated the phrasings I expected ("four items below", "four open obligations", ...) rather than the concept. A concept search (`\bfour\b` with context) finds it immediately and confirms it is the ONLY stale one: every other "four" is the genuine four-row table. Stand: Echo Act IV Pro * docs(pool): the claim proves ownership, not execution; presence is not completion keweichen, REQUEST_CHANGES at this exact head. Both blockers reproduced against the cited authorities before changing anything — he is right on both. **1 — a `sole source` table assigned a DERIVED decision to one partial input.** The claim row named the hard link as the sole source for "may this task be executed". The parent contract says the opposite at :1094-1139: "Claim first, then RE-READ the bindings, and release without executing if the room is no longer..." First-pin is deliberately unfenced, so a claim can still name an instance the room no longer binds. Split into what the hard link actually proves (exclusive ownership) and a DERIVED row for execution authorization naming the post-claim re-read. The eligibility row had the same shape and is now stated as a conjunction over its separately owned inputs (eligibility, probation, bindings, .alive, the quiesce exclusion) rather than assigned to two of them. **2 — result-file presence is not completion.** `src/delivery/readiness.py`: "File existence is therefore not readiness: a consumer that treats it as readiness delivers an empty message and archives the task as done, which strands the real answer written moments later." The completion row and the crash-predicate list both canonized presence. Both now name `read_ready_result`. Step 2's implementation scope names `task-notifier.sh::has_result`, which uses a raw `[ -f ]` at :102-107,187 — the runtime fix stays in that implementation PR. **Pinned, because he measured that deleting these rows left all 128 tests green.** Mutation controls, each run against the corrected tree: baseline rc=0 M1 revert claim row to "may this be executed" rc=1 M2 revert completion row to file presence rc=1 M3 revert crash predicate to "presence" rc=1 M4 DELETE the derived-execution row rc=0 <- unpinned M4 after adding the existence assertion rc=1 M4 caught my own pin: it constrained only rows that exist and never asserted the execution row must exist, which is the same defect one layer along. Fixed. retractions 133 tests OK (was 128) · transitions 84 OK · review-checks PASS Stand: Echo Act IV Pro * docs(pool): the claim decision reads the task's address, not instance state alone The authoritative-source table derived "may that instance claim right now" from a conjunction over instance state only. Rule 1 (:778-790) reads `requested_worker` first, so two tasks differing only in that field decide differently on identical instance state — the listed inputs cannot answer the question they claimed. Split into three rows: the operational conjunction (declared BLIND to the task, bindings.json removed from it), the task-level address, and the ordered per-task decision. :2134 no longer assigns the per-task question to the eligibility reader, which left two decompositions of one question standing at once. Their control — deleting the row — left ran=133 failures=0. Now: M1 delete the per-task row FAILED (4) M2 strip every requested_worker FAILED (1) M3 remove the ORDERED qualifier FAILED (1) M4 restore the old decomposition FAILED (2) baseline / restored OK (139) One pre-existing test pinned the old wording; updated the string, kept the assertion. Reported-by: keweichen Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the claim derivation spans all three routes, and addressing has two forms The replacement row named rule 1 and the pin-table half of rule 2, cited `:778-790`, and stopped immediately before rule 3 — so it could not answer the unbound-room case where the core claims and workers suppress. It also called `requested_worker` the sole answer to what a task addresses, contradicting `:104-115`, which makes the envelope field and the pin table both forms of addressing. Address row now names both forms. Decision row spans `:778-796`, all three routes. Also repaired a test of my own that passed for the wrong reason: `assertIn(":778-790")` was satisfied by the row's own warning AGAINST citing that shorter range — a negation standing in for the citation. It now pins `:778-796`, plus two coverage assertions. M1 cite the short range again FAILED (failures=1) M2 drop rule 3 from the row FAILED (failures=1) M3 drop the pin table FAILED (failures=1) baseline / restored OK (141) Reported-by: keweichen Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the addressing row is derived and ordered, not a second authority The authoritative-source table promises each artifact answers one question and labels the second column `sole source`. The addressing row broke both: it reused `bindings.json`, already the sole source of room membership one row above, as a co-equal answer to what a task addresses, and unlike every other multi-input row it was neither marked DERIVED nor ordered. That left the conflicting-input case undecided in the document while the contract decides it: with `requested_worker: worker-2` in a room pinned to `worker-3`, rule 1 has worker-2 claim and worker-3 suppress, but the table said field and pin both answer, with no order. Mark the row DERIVED and state the order — the envelope field outranks the pin table — so the table agrees with rule 1 and `bindings.json` stays the sole source of membership only. Stand: Echo Act IV Pro * test(pool): pin the addressing row's ORDER, not just its two inputs keweichen ran the mutants I should have: reversing the row's stated precedence, and deleting rule 2's dedicated-room branch, both left retractions 141 OK and transitions 84 OK. The existing address-row test asserts only that the row CONTAINS `requested_worker` and `bindings.json`, so the previous commit changed one documentation line that no test could tell from its opposite. Pin the direction and the DERIVED marking. Mutants, run at this head: reverse precedence (pin outranks field) FAILED (failures=1) drop the DERIVED marking FAILED (failures=1) drop OUTRANKS (names both, orders neither -- the original defect) FAILED (failures=1) baseline OK This closes their third finding only. Their first two are contract changes, not test gaps, and are answered on the PR. Stand: Echo Act IV Pro * docs(pool): gate the routing split instead of canonizing it keweichen and qingyun-wu both flagged two cumulative routing contradictions at 2dd8285. Neither is created here; both are inherited from the parent and this layer stated them as settled in the authoritative-source table. Selection and post-claim authorization use different authorities: rule 1 lets requested_worker outrank the pin table, while the post-claim re-read authorizes on membership alone. A pool command addressed requested_worker: core in a pinned room wins the claim and then releases, and rule 1 has suppressed everyone else. The dedicated own-room route names no artifact proving the relationship. Resolving the split would mean deciding the pool overrides a SHIPPED contract: tests/requested-worker-header.test.py says the field is intent, not placement, and no claim path reads it -- measurably so, requested_worker occurs in exactly two files under src/, both header-key declarations. That is a design decision above a docs change, so both are gated on step 2 and left to the step that implements them. Four mutation controls, each failing exactly one test: removing either gate entry, unqualifying the addressing row, or dropping MEMBERSHIP ONLY from the EXECUTE row. 143 -> 147 tests. Stand: Echo Act IV Pro --------- Co-authored-by: Chi Wang <sonichi@users.noreply.github.com> Co-authored-by: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ehind an adapter, one transactional task store (supersedes the #3860 draft) (#4037) * docs(pool): worker-pool design v1 — one local supervisor, executors behind an adapter, one transactional task store The normative v1 worker-pool design, restructured around the owner's 2026-09-07 review of the #3860 draft: a single Sutando pool supervisor that depends on no LLM session is the only scheduler; core and workers are executors behind Executor.offer/accept/start/cancel/events/health with Claude, Codex App Server and ACP adapters; task control state is one SQLite table with PENDING -> OFFERED -> ACCEPTED -> RUNNING -> SUCCEEDED | FAILED transitions as single conditional updates; explicit accept plus lease, supervisor-owned reconciliation of expired leases; room-to-worker binding with requested_worker override; bound-but- unavailable rooms stay PENDING with Rebind / Restart / Process with core surfaced to the owner; a four-state health model with a probing transition on kick. The record behind it — rejected alternatives, the failure model taken from the #3860 draft, and the owner decisions — lives in a separate notes file, so the normative text carries no review history. Tests: a retraction suite that rejects the superseded mechanisms (lead or router in the daemon, core as scheduler, watcher-owned admission, filename-keyed claims, per-worker proactive loop, the file-protocol directories, TASK_FILE as the delivery abstraction) each with a live positive control, and a transitions model of the store's state machine with a lease clock, crash seams, supervisor restart and the four crash windows. * test(pool): the transitions model's sweep is reconcile_leases, not reconcile The delivery-core contract scanner treats every function named `reconcile` under tests/ as an adapter of that contract and requires the `attempt` signature. The model's lease sweep is unrelated to it, so it takes a name the scanner does not claim. * docs(pool): the store is a supervisor-owned file journal, not SQLite Per the owner's 2026-09-07 decision: the authoritative task state is not a database. Each task keeps an immutable payload `tasks/<id>.txt` and a per-task `task-state/<id>/state.json` that the supervisor — the only writer — replaces atomically (temp + fsync + os.replace + fsync dir). Executors never write authoritative state; they report over a Unix socket and leave a durable replayable receipt under `executor-events/<executor>/`, which the supervisor consumes on restart. `lease_generation` rejects a stale completion into `stale-results/`; `bindings/rooms.json` is the supervisor-written pin table; a single instance is guarded by an advisory `flock`, never a PID file. Recovery scans non-terminal state records and unconsumed receipts. One authoritative source per fact — no state expressed through filenames. Replaces the previous head's single SQLite `tasks` table and its conditional-UPDATE transitions. The record of why (and the two codex-branch nudge defects surfaced while diagnosing the pool) moves to the notes file; the normative doc carries no history. Retraction suite gains rejections for a marker-file state source, a claimed-suffix rename, an executor writing state.json, a PID-file instance guard, and SQLite as the store. The transitions model now covers the atomic-replace seam, the generation check with stale-complete archiving, receipt-inbox restart replay, and the single-instance lock. Note: bindings/rooms.json keeps an unordered instance set + pinned flag (v1's routing semantics) rather than the decision memo's single-executor shape; flagged in the PR for the owner. * docs(pool): drop the rejected-SQLite table and non-English quotes from the record The notes recorded the owner decisions as verbatim non-English quotes with English glosses, and documented the superseded store as a SQLite table with its DDL. Keep the English renderings only, and describe the replaced store as a generic database-backed store without the table schema. The normative doc's out-of-scope line no longer names SQLite. No rule or mechanism changes; the journal remains the store and every guard test is unchanged. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pool): address review — durable delivery intent, state-specific expiry; carry remaining #3860 coverage Review (pullrequestreview-5138901924): - Row 6 now writes the result and the outbox record (delivery intent, keyed on the task id) before clearing the lease identity, so a crash in the seam re-drives from the outbox and never drops a completed reply. Recovery re-drives outbox records without a delivered sentinel. - Expiry is state-specific: an OFFERED record expires on offer_expires_at, an ACCEPTED/RUNNING one on lease_until. Fixes the reconciliation step and the non-terminal invariant that named only lease_until, which would strand an offer no executor accepted. The transitions model now carries offer_expires_at as a distinct field and adds two crash-seam tests (offer expiry; a completed reply surviving the identity clear). - The terminal owner-decision note is reframed as the design's own rationale rather than an unverifiable quotation of a private discussion. #3860 coverage carried so #4037 fully supersedes it: - the per-task timing log data/pool-metrics.jsonl as the saturation substrate; - auth (401 -> recycle) and transport (5xx -> back off) worker-failure handling; - the pin lookup keying on the provider-native room id (channel_id / chat_id). Guard tests: retractions 14, transitions 22, docs-audit 5. No model or rule regressions; the journal remains the store. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pool): record the lessons from real traffic in the notes Three failure modes seen running the lead-follower pool, each removed by construction in this design rather than left to be tuned: - the no-claim cooldown equal to the stuck window (NOCLAIM_COOLDOWN_S == ASSIGN_STUCK_S), a guard whose failing case is unreachable — the single-offer generation lease has no cooldown to tune; - keystroke-into-a-TUI nudging (staged-entry phantom success, an ANSI-misread prompt, a plist/pane runtime mismatch), replaced by the executor adapter's accepted offer with an echoed assignment_id; - the per-worker /proactive-loop-pool pass sweep, replaced by the supervisor's event-driven offer plus its periodic reconciliation backstop. Record-only (notes); no normative rule or model change. Guard tests unchanged: retractions 14, transitions 22, docs-audit 5. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * docs(pool): lead with the PR #3860 requirements, verbatim The owner asked for #3860's requirements at the top of the design doc in exact words. Reproduced verbatim, ahead of the design, with the source PR cited. Two conflicts with the design below are flagged and left UNRECONCILED for the owner rather than merged silently: - the requirements name a router installed with worker 1; the design names a pool supervisor that is the only scheduler, and #3860's own title says 'no router process' - the requirements are dated owner-decided 2026-09-03, the design 2026-09-07 Nothing in the existing design was altered; the file is prepend-only. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): record the #3860-vs-design conflicts in a Discussion section The owner asked that conflicting inputs be marked in a discussion section rather than left as an inline aside. Four substantive conflicts, each with both sides quoted: 1. task files renamed (requirements) vs immutable (design) — the two on-disk protocols are not implementable together 2. atomic rename + done-flags vs lease + generation as the at-most-once primitive 3. whether a scheduler exists before the first worker 4. whether workers keep draining while the scheduler is down Naming (router vs pool supervisor) follows from those, so it is not settled separately. What is NOT in dispute is listed too, so the open question stays as small as it actually is. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the assignment is a delivery record, not a rename of the task Rewrites the normative design and both guards around the owner's 2026-09-08/09 decisions, and answers the two P1s on this PR. Content of the assignment moves out of the task filename. One immutable payload lives at tasks/<id>.json; each recipient gets a zero-byte sentinel at deliveries/<recipient>/<id> whose existence IS the assignment, renamed to .claimed to claim it. A set is one payload and N sentinels, so N recipients no longer means N copies that can drift. Also settled here: the router exists from the first install and rides in the hardened task watcher rather than a daemon of its own; the core reads deliveries/core/ always, so no protocol switch fires at the zero-worker boundary; the process supervisor replaces launchd as the role, and never supervises a worker's session; worker id and label are separate, so a rename is a one-field edit; every task worker has a delivery mechanism, which in Sutando is a watcher; workers may run their own crons, whose output passes admission as a request. P1 (authorisation reduced to envelope integrity): authorisation is resolved, never read off the stamp, and every pool command needs an independently resolved owner capability. P1 (command ownership): the core holds lifecycle and spend, the router only places, and no control action's mechanical path runs through an LLM session. The retractions guard previously required the filename protocol it now rejects. Rewritten with 11 rejections; red/green measured — 7 fire against the previous revision of the doc, 0 against this one. The transitions suite modelled the lease-and-journal design and is replaced by a model of payload plus sentinel on a real filesystem, 28 tests, including one asserting a set writes exactly one payload per branch and zero-byte sentinels. Adds an implementation section: layout and JSON shapes, the router pass with O_EXCL and EEXIST-is-success, the worker loop and boot sweep, finish's refusal conditions and write ordering, a four-case residue table, the supervision timer's two signals, per-stage acceptance criteria, and the dual-intake migration rule. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(pool): close six unchecked design claims, and fix the double-delivery they exposed Audited the transitions suite against the doc: the core protocol was covered, six stated claims were not. Adding them found a real gap in the router pass. **The gap.** Step 4 said a delivery already present makes the pass idempotent, checking only `deliveries/<t>/<id>`. A claim renames that sentinel to `<id>.claimed`, so the unclaimed name is free again and a second pass recreates it — delivering work already in flight a second time. The pass now treats a sentinel under either name as delivered, and the spec says why. Added, each mutation-checked (remove the behaviour, the test must fail): - a repeated pass delivers once, including after a claim - archiving never clobbers an existing record (.1, .2) - priority orders before age - a sentinel whose payload is gone is removed - a stall and a death are different signals, and only one authorises a restart - creating a worker creates its delivery mechanism, removing it removes it Six of the seven mutations are detected. The seventh, tolerating EEXIST, is unreachable single-threaded now that the claimed-aware check precedes it: it guards a concurrent second pass, which this model cannot stage. 36 tests, up from 28. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): step 1 of #3604 with no luggage, and reject a second claim protocol Owner decision 2026-09-09: still step 1 of staging #3604 into PRs against main — the plan is #3604's, the code is not. Implementation targets main directly and starts with no luggage; the rescue line is the reference implementation, never a base. A section records what that costs: it diverges while Stage 1 lands, and retiring the hosts running it is a separate change. The guard gains a rejection it was missing. `watch-tasks-stream.sh` already has an extension point, SUTANDO_TASK_EVENT_HANDLER, and setting it starts a second claim protocol — state/task-event-handler-claims, -fallbacks, and a pending/running/ settled dispatch queue, with acquire/release/retire claim helpers. That is one fact expressed through several files, which this design replaces with one atomically renamed record. Nothing in the guard would have caught a revision that described it, so rejection 9 now does. Verified both ways: it fires on those state directory names and stays silent on this doc. The hook itself is fine and unset everywhere outside Codex's launcher; the protocol behind it is what must not come along. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the previous pool is stopped, so Stage 1 starts on a clear field The rescue-line section said retiring the hosts running it would be a separate change sequenced after Stage 1 proves out. That is now backwards: both cores were stopped on 2026-09-09, before Stage 1 began. Verified before stopping them — assignments.json held zero entries, no task file carried a claimed or assigned suffix, and the queue was empty, so nothing was in flight. Stopping took three steps because the processes had drifted away from the jobs that nominally owned them: booting out com.sutando.core-1/-2 left every process alive under tmux, killing the two tmux sessions left two more as children of a leftover server, and those needed a direct SIGTERM. Same shape as the Conflict 5 measurement — a launchd job proves nothing about its process tree. The section now records what remains rather than what was planned: dead state/pool/ (affinity.json, assignments.json and siblings) which Stage 1 neither reads nor migrates, with its removal a separate cleanup. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the worker id is minted, never chosen — and never reused The document already required the id to be "opaque, never changed or reused", then used `worker-2` as an id in every example: the delivery path, the roster key, the binding values and the payload's `requested_worker`. A sequential form cannot satisfy the rule it was illustrating. Two ways it fails. "Next free N" hands a deleted worker's id to its successor, so a surviving sentinel, `.alive` file, done-flag or archived header is read as belonging to a stranger. And two hosts sharing a synced workspace both compute the same next N, neither wrong locally. Adds the field table the owner specified (worker_id / label / incarnation_id), the minting rule — `worker-` plus 6 lowercase hex, atomically registered so a duplicate is refused loudly at create time rather than silently aliasing two workers — and the lifecycle rules that give the constraint its purpose: * recovering, restarting or changing session KEEPS the worker_id; * every new session mints a fresh incarnation_id; * delete-then-create mints a NEW id even when the label is reused. Owner chose the short hex form over a bare uuid4().hex: 32 hex characters is exactly the `[a-z0-9][a-z0-9-]{0,31}` ceiling, leaving no room for the `worker-` prefix that makes a path readable. Every id example now uses a minted form; `worker-1` survives only as a LABEL, which is what it should have been throughout. Design guards unchanged and green: retractions 7/7, transitions 36/36. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): the worker id is uuid4().hex — entropy, not a short suffix Corrects three things in the previous commit, all raised by the owner. The random part was too short. `worker-` plus 6 hex is 24 bits: 1,000 independently minted ids collide with probability 2.9%, and 95% by 10,000 (verified: 2.9363% and 94.9219%). The prefix contributes readability, not entropy. uuid4 carries 122 random bits, which is what makes generation safe without a coordinator, and its 32-hex form is exactly the `[a-z0-9][a-z0-9-]{0,31}` ceiling. Opaque was conflated with unguessable. Opacity is a demand on the READER — treat the id as one value, never parse role, order or meaning from it. Uniqueness and non-reuse are separate requirements met by the generator. Unguessability is a consequence here, not the goal. The counter was rejected for the wrong reason. Persisting the highest allocated value would prevent recycling without remembering every id ever issued; the actual defect is allocation ACROSS HOSTS, where two machines on a synced workspace compute the same next value and neither is wrong locally. Adds the rule the previous version left out: atomic registration solves LOCAL concurrency only. When a cross-host sync surfaces two workers claiming one id it must fail and report — never overwrite, never silently merge, since both destroy the identity the rule exists to protect. Design guards green: retractions 7/7, transitions 36/36. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(pool): assignment is a rename into the recipient's folder, and the router asks one question Owner decision 2026-09-09: adopt the rescue line's rename-based assignment, but rename INTO deliveries/<recipient>/ rather than suffixing in tasks/. The payload moves; where the file is, is who owns it. This replaces the zero-byte sentinel beside an immutable payload, which was two encodings of one fact -- two records that can disagree -- and drops the orphan-delivery and vanishing-payload failure modes with it. A set is the one case that copies, member list persisted first. The router now asks only whether every target is on the roster. Unknown names go to the core, which is a recipient rather than a fallback; an addressed worker is never substituted by another worker. Liveness is not read: a delivery is a file in a folder and a worker that starts later finds it. Holding was rejected because a held task existed only inside one router pass, recorded nowhere. The worker-state table is now the core health-check's, not the router's. requested_worker is honoured; the gateway maps the broker's target_worker onto it at the boundary. The worker loop is acquire_work with its unassigned-pool fallback removed -- that fallback was routing by another name. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): crash-safe retirement as an idempotent sweep; one fact, one record Two requirements the #3860 coverage check found uncovered. Retirement touches two records -- roster and folder -- and no two file writes are atomic together, so it is ordered and every step idempotent: roster first (the only decision), then a per-tick sweep moving the folder's .txt back to tasks/ and leaving .claimed to finish or release, then rmdir only when empty. Nothing under deliveries/ is deleted unread. A router pass racing the retire strands nothing: the next tick sweeps it. The partial states reproduced against the rescue line cannot arise: there is no token, no gate directory, no second record -- the folder's contents are the state. A table of every question with its one authoritative record and what it must never be inferred from. A second answering record is a defect while the two still agree; three review rounds on the rescue line found that class. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): the delivery is a zero-byte sentinel; the payload never moves until finish Reverses the previous commit's move-the-payload model on the owner's decision, and records why so it is not re-argued: a set to N workers under the move costs N payload copies plus a persisted member list to survive a crash mid-copy; under sentinels it is N empty files and one payload. The "two encodings of one fact" objection that motivated the move applies to the rescue line's rename-in-tasks/ and to running both schemes together -- not to a sentinel beside an immutable payload, where the sentinel's folder is the single ownership record and the payload's location records nothing. Everything else from tonight stands: per-worker folders, .txt naming so the watcher wakes, one-question router with no liveness and no hold, crash-safe retirement as a sweep, one fact one record. The two sentinel residue rows return. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): keep #4037 to design decisions; retirement ordering and the per-question record table move to the long-form artifact Owner call: both are implementation-level. They are recorded as a note in the 'Durable Worker Pool in Sutando' artifact instead. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): six inconsistencies found by worker 1's alignment review A done-flag with no result is terminal: the bridge drains results/<id>.txt on delivery, so after a drain the flag is the only evidence left; reading that as died-mid-work re-runs finished work. New residue row. The payload is the bridge's header-format tasks/<id>.txt, not JSON; layout, archive naming and the payload block now agree. Binding keys are the bare channel id the envelope carries. A sentinel is empty, so no assignment carries a roster version. The router writes no pool-status.json, so the layout no longer lists one. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): the recipient accepts work; it never claims it Owner decision 2026-09-10. 'Claim' was the rescue line's word for a core selecting from an unassigned pool; here the folder already decided the recipient, so the rename records acceptance of work assigned to it. .claimed -> .accepted throughout, the section becomes 'Request, assignment, acceptance', and the pending state is 'pending'. Mechanism unchanged. The long-form artifact already carries this word. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): finish the accept rename in prose Eight prose uses of 'claim' for the worker's own step. The two that remain are the OTHER sense — a worker selecting unassigned work, and two hosts asserting one id — and the design forbids the first, so the word is doing real work there. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): the core's intake move is deferred, with a revisit trigger Owner decision 2026-09-10, relayed through the live pool worker: keep the current core intake for this release and prioritise stability. The section already permitted preserving today's behaviour for the first pool release; this records that the option was TAKEN, so the design and the plan in #4105 (FLAG-1) do not drift. Trigger written down rather than left to memory: a second worker exists, or the first feature that must cover both core and worker is specified. The accepted cost is two intake protocols. Co-Authored-By: Claude Fable 5.1 <noreply@anthropic.com> * docs(pool): the core's first-release intake matches FLAG-1, and the index entries describe the built design The intake paragraph said the core "always reads deliveries/core/" while FLAG-1 defers exactly that move; the README and catalog entries still advertised the superseded supervisor/journal design. * test(pool-design): the retraction guard rejects the mechanism, not the words Written by worker 2 (Sudoo-the-sutando-dev); committed on the owner's "commit everything" instruction. REJECTED[6] now rejects fan_out, or target_worker followed or preceded within a sentence by honour / obey / decide / select / choose / route, so the boundary-mapping sentence is exempt; the 'renaming it claims' control re-keys to the invariant wording; a new class shows the exemption comes from the pattern, not denial words. * docs(pool): the recovery policy — the core brings a worker back; moving its task is the owner's call Owner-settled after tonight's incident, in which the core answered a worker's task in the worker's room: diagnose the cause and take the fitting action (relaunch a crashed session, wait out an exhausted quota window); keep the task's owner, context and progress meanwhile; offline, timeout or a failed recovery never authorise a takeover; ask the owner with cause, attempts, expected recovery and affected tasks; after approval, an explicit handoff that the original worker is told about. --------- Co-authored-by: qingyun-wu <qingyun@ag2.ai> Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
What
docs/worker-pool-design.md: the v1 router design for the worker pool, as decided by the owner on 2026-09-03 in the PR-triage room. Step 1 of staging #3604 into PRs againstmain; #3604 stays open as the reference implementation. Docs only, one new file, no code.The decisions it records (owner, quoted)
What it supersedes, and why that is stated
docs/lead-follower-pool.mdplaces the lead inside the runtime daemon (src/runtime-api/server.py). Measured at 6c0b416:server.pyhas zero references to the lead, the daemon is not started by any launcher on this host, and the doc's only reason for that placement (a single admission point) was never built. The router is its own process and the doc now says so.docs/core-pool-standing-sessions.md("fixed N is replaced by lead-managed sizing") is replaced by core-on-owner's-word sizing; the other decisions in that record stand and are referenced.Evidence
Everything measured is measured against #3604 at 6c0b416 and named in the doc: the five-tier
_pick()collapsed to two cases, the header override removed, the--pool-maxauto-scale branch excluded, the per-config-dir model limitation named as the installer PR's gap. The coordination contract (rename assign / claim, done-flags, 30 s / 90 s beats, three reclaims, leaderless degraded mode) is carried over unchanged from #3604.Stand: Echo Act IV Pro
Update d0d5531 (owner, 2026-09-03 19:5x PDT): v1 has no router process. With placement fixed by the pin table, priority is a local sort per claimant and the router's only remaining duties are liveness, which the core owns (sweep, reclaim behind the done-flag, stand-in for a pinned worker whose beat is stale, kickstart, status line). Assignment and degraded mode leave the coordination contract; workers claim only. A router earns a process only when a whole-queue policy arrives, and those are all out of v1. Staged PRs re-cut accordingly: doc; worker claim path + pin table; core sweep; installer.