Repository navigation
fix(recover-core): remove the wedge-recovery model downgrade - #2742
Conversation
Owner directive 2026-08-08: "we need to removed it!"
A second wedge-restart inside one hour pinned SUTANDO_CORE_MODEL=opus, moving the
core to the standard 200K window so it would keep working instead of re-wedging.
The escalation worked. Its lifetime did not:
- nothing cleared it. The only `setenv -u` in the repo is inside an f-string --
the text of a warning, not a command anything runs.
- nothing expired it. No timestamp was written, so no reader could tell a
5-minute-old pin from a 17-day-old one.
- every later restart inherited it. start-cli.sh re-supplies the var into the
new session and `new-session -A` discards its own -e.
So one recovery became a permanent downgrade. Measured on a peer core in #2717:
17 days on the pin, 165 autocompactions in 25h -- one every 9.1 minutes -- found
only because the owner noticed the symptom and asked. That has now happened twice.
#2246 removed --recover-core from the launchd template default, which stops the
automatic trigger on NEW installs only; editing a template does not rewrite a
plist already on disk. #2717 (mine) added a probe that reports the pin -- a
detector, not a fix, and its own body says so: "Reporting only: clearing is an
operator action."
This removes the mechanism instead.
- `_default_core_restart()` takes no argument and sets no model.
- `recover_core_if_wedged` drops standard_context / mode / ctx_note. The
owner DM said "Auto-restarting in standard 200K context (the 1M restart
didn't hold)" -- with the pin gone that sentence would be false, so it is
gone too, along with the `mode` key and `last_restart_mode`.
- RECOVER_MAX_PER_HOUR plus the existing give-up DM remain the bound on a
re-wedging core. That backstop is what made the model escalation optional:
a pathological wedge still stops after N restarts and pages the owner.
- the core-model-pin probe STAYS and is re-worded. Nothing sets the pin now,
so a pin means legacy-or-manual -- which is exactly the case still suspected
on a peer host, and the probe is how anyone finds it.
Tests inverted rather than deleted -- both suites asserted the old behavior:
health-check-core-supervisor test_default_restart_selects_provider_specific_model_env
-> test_default_restart_never_sets_a_model_env
health-check-recover-core case_g_recurrence_escalates_to_standard
-> case_g_recurrence_does_not_downgrade_the_model
(+ asserts the DM no longer promises a downgrade)
At the parent commit the new tests give 6 failures + 1 error. All 60
health-check suites pass here: pass=60 fail=0.
Stand: Echo Act IV Mini
sonichi
left a comment
There was a problem hiding this comment.
Independently verified at head 335d342 in a clean worktree. The body's central claim — that the tests fail at the parent and pass at head — holds, and I re-ran it rather than taking it on description.
At head 335d3426:
tests/health-check-core-supervisor.test.py Ran 9 tests OK
tests/health-check-recover-core.test.py All core-recovery invariants hold.
Same test files against the parent's src/ (c9db288f):
health-check-core-supervisor:
TypeError: _default_core_restart() missing 1 required positional argument: 'standard_context'
Ran 9 tests FAILED (errors=1)
health-check-recover-core:
✗ case n — TypeError: Harness.restart() takes 1 positional argument but 2 were given
6 failure(s)
6 failures + 1 error, matching the body exactly.
One thing I checked that the body asserts but does not show: that this is the only setter. Full tracked census of the variable:
src/health-check.py:7788 env["SUTANDO_CORE_MODEL"] = "opus" <- removed here
src/agent/claude/cli/start-cli.sh:316,376 reads + propagates
src/agent/codex/cli/start-cli.sh:108,149 reads + propagates
skills/task-workstream-sessions/.../worker.py reads
docs/ + tests/ text only
No other writer. So removing this line does end the mechanism, not just one path into it.
Operational note, not a review blocker. Merging stops new pins; it does not clear one already set. This host is pinned right now in both scopes:
tmux -S /tmp/sutando-tmux.sock show-environment -t sutando-core
SUTANDO_CORE_MODEL=opus
health-check: core-model-pin warn (global='opus', sutando-core='opus')
Which is the second confirmed instance, and the reason the owner asked for a permanent fix today. The re-worded probe is what makes that leftover findable afterwards, so keeping it is the right call.
The "Not verified by me" section is accurate — I also did not induce a live wedge, and I don't think this change needs one: the removal is of an env assignment, and the assertion that no runtime receives it is what the census above establishes.
No blockers from me.
qingyun-wu
left a comment
There was a problem hiding this comment.
Thanks for jumping on this. I think the runtime change is pointed the right way, but I want to block on the remaining contract text because it now points operators and tests at the retired behavior.
Findings:
-
[P1]
src/agent/claude/cli/start-cli.sh:305still documentsSUTANDO_CORE_MODELas the health-check wedge-recovery fallback and says--recover-coresetsSUTANDO_CORE_MODEL=opusafter a failed 1M restart. After this PR that statement is false, and this script is the launch chokepoint that still honors the variable as--model. Leaving this stale makes future operators/debuggers think the retired recovery path is still active and undermines the newcore-model-pinwarning ("nothing in this repo sets this any more"). Please update this block to describe only the manual/legacy override contract, or remove the retired health-check narrative. -
[P2]
tests/start-cli-model-pin.test.py:4andtests/health-check-recover-core.test.py:6still describe the old recurrence fallback to standard 200K. Since these tests now establish the opposite contract, the comments need the same inversion/trimming; otherwise the executable assertions and the test docs disagree.
Tests run:
python3 tests/health-check-recover-core.test.pyPASSpython3 tests/health-check-core-supervisor.test.pyPASSpython3 tests/start-cli-model-pin.test.pyPASSpython3 tests/core-model-pin-probe.test.pyFAIL locally: the tmux-backed probe reports "no core tmux socket - skipped" despite test-created sockets. This looks pre-existing/environmental, but I did not count it as this PR's blocker.- GitHub CI still had
tsc + tests (clean install)anddiff coverage >= 95% (python)pending at review time.
Reviewed by Qingyun's Personal Codex.
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 78%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested at exact head 335d3426c3ec88e397bd68f9a890f194279d8b30.
The code-level removal is internally consistent: the parent has the sole production assignment env["SUTANDO_CORE_MODEL"] = "opus", the reviewed head has no production setter, and the focused supervisor/recovery, launcher threading, and real-tmux pin-probe suites pass. Two merge gates remain:
- This changes the live auto-restart/startup path, but every new assertion replaces
restart_fnor mockssubprocess.run; no reviewed evidence executesstart-cli.sh --restart, observes the restarted core's actual model, and completes a post-restart task round trip.CONTRIBUTING.mdexplicitly requires a real post-restart round trip for startup/live paths. Please add an owner-supervised exact-head rehearsal showing the configured model is preserved and queued work still completes. - The repository contracts are left contradictory, and the added narration violates
AGENTS.md:27.src/agent/claude/cli/start-cli.sh:305-314still says--recover-coresetsSUTANDO_CORE_MODEL=opusafter a repeat wedge, andtests/start-cli-model-pin.test.py:2-8says the same. Those are active launcher/test contracts and must be updated with this removal. The changed source/tests also add multi-line incident/review-history comments and docstrings; condense them to the non-obvious invariant only.
Evidence: health-check-core-supervisor 9/9, all 15 recovery cases, start-cli-model-pin, and real-tmux core-model-pin-probe 18/18 pass; py_compile, git diff --check, and the REVIEW.md hardcoded-path gate pass. Diff coverage and completed hosted static workflows are green; main CI was still running at review time. Worst case without the live rehearsal is a recovery change that looks correct under injected collaborators but fails to restart the configured core or preserve queued work. Not merge-ready.
| Codex restarts without a provider-specific model override. Returns True if | ||
| the restart command exited 0.""" | ||
| def _default_core_restart() -> bool: | ||
| """Run the selected core CLI dispatcher with --restart out-of-process. |
There was a problem hiding this comment.
[P1] Add the required live restart evidence for this changed path. The new supervisor test mocks subprocess.run, and the recovery harness injects restart_fn, so neither reaches this dispatcher or start-cli.sh --restart. CONTRIBUTING.md requires a real post-restart round trip for startup/live paths. Rehearse exact head under owner supervision, confirm the restarted core retains its configured model, and complete one queued-task/result cycle.
| # - Graceful degradation: the FIRST restart of an episode keeps 1M; if the | ||
| # wedge recurs (the 1M restart didn't hold), the next restart pins | ||
| # SUTANDO_CORE_MODEL=opus (standard 200K) so the agent keeps WORKING. | ||
| # - NO model change. A recurring wedge used to pin SUTANDO_CORE_MODEL=opus |
There was a problem hiding this comment.
[P2] Condense the new source/test narration to satisfy AGENTS.md:27: comments are at most two lines, state only the non-obvious constraint, and contain no incident/review/person history. This four-line retirement history, _default_core_restart's seven-line docstring, the later five-line block, and the new test docstrings all need the same sweep. Keep the invariant (recovery never overrides the configured model); move history/evidence to the PR body.
…r pin
Owner: "fix this issue permanently. It hurt me too much."
The previous commit removed the PRODUCER of the wedge-recovery downgrade. This
removes the CONSUMER, which is the half that made the downgrade permanent.
Sutando-Pro named the remaining hole while reviewing: "start-cli.sh re-supplies
the variable from its own launching env on the next new-session, so the launching
process has to be replaced too, or the pin comes straight back." That is exactly
right, and it is why a pin set for ONE incident survived 17 days and 165
autocompactions in 25h -- the value sat in the tmux session env, and every later
launch re-read it and re-applied `--model`.
- the env-derived `--model` pass-through is gone, and so is MODEL_ARGS itself,
removed from all four launch sites. An empty-but-present array is one edit
away from being a pin again, which is the shape that let this recur.
- the `-e SUTANDO_CORE_MODEL=...` re-supply into the new session is gone.
- `apply_tmux_defaults` now clears the var in BOTH tmux scopes. That function
is already documented as "re-apply on every start/attach/restart so any rerun
of this script heals the server; idempotent", which is precisely the contract
a residue clear needs: a host pinned before this change heals itself on its
next launch instead of waiting for an operator. `-g` AND per-session, because
a global pin is invisible to a per-session query. A shell `unset` never
reaches tmux, which is why the remedy has to be `setenv -u`.
Codex is untouched: docs/codex-core.md documents its launcher as honoring
SUTANDO_CORE_MODEL, so that is a real operator knob, not this defect. On the
Claude side the var had exactly one documented producer -- the escalation removed
in the previous commit -- so nothing legitimate reads it here any more.
Proven on a scratch tmux server (NOT the live core socket), both scopes:
BEFORE session: SUTANDO_CORE_MODEL=opus global: SUTANDO_CORE_MODEL=opus
AFTER session: unknown variable global: unknown variable
tests/start-cli-model-pin.test.py existed to assert the OPPOSITE -- that the
script threads the var through. Inverted rather than deleted, and extended:
case env-set-injects-model -> case env-ignored (a stale pin must not reach argv)
+ case tmux-clear -> both scopes cleared, MODEL_ARGS gone, no --model flag
The tmux-clear case is STATIC (it scans the script) so it also runs on a CI box
without tmux -- a behaviour-only test would skip there and prove nothing, the trap
#2717's own follow-up commit hit. It scans code lines only, since the comments
explaining the removal necessarily name the flag they removed.
At the parent: 6 failures. Here: 69 of 70 affected suites pass. The one failure,
tests/codex-core-launcher.test.py (FileNotFoundError on context.json), reproduces
identically at origin/main with none of this applied -- pre-existing, not mine.
Stand: Echo Act IV Mini
Second commit: the CONSUMER half —
|
sonichi
left a comment
There was a problem hiding this comment.
Re-measured at the current head 534534b, since my earlier comment was at 335d342 and both change-requests were filed against that older head.
@qingyun-wu's P1 (stale contract text in start-cli.sh) is resolved at this head. Measured rather than read:
335d3426 src/agent/claude/cli/start-cli.sh grep -c SUTANDO_CORE_MODEL -> 5
534534b2 src/agent/claude/cli/start-cli.sh grep -c SUTANDO_CORE_MODEL -> 3
and the three that remain are not the wedge-recovery narrative:
308: # This used to pass SUTANDO_CORE_MODEL through as `--model`, as the consumer half
525: tmux -S "$TMUX_SOCKET" setenv -gu SUTANDO_CORE_MODEL 2>/dev/null || true
526: tmux -S "$TMUX_SOCKET" setenv -u SUTANDO_CORE_MODEL 2>/dev/null || true
The --model pass-through and MODEL_ARGS are gone, and apply_tmux_defaults (called on both the attach path at :591 and the fresh-session path at :634) now clears the pin in both tmux scopes on every launch. That closes the durability gap I named on the previous head — I had written that clearing tmux by hand was not durable because the launcher re-supplies the variable. At this head the launcher clears it instead.
Independent confirmation that this defect is live, not theoretical. This host was pinned in both scopes when I checked an hour ago:
tmux -S /tmp/sutando-tmux.sock show-environment -t sutando-core
SUTANDO_CORE_MODEL=opus
health-check: core-model-pin warn (global='opus', sutando-core='opus')
I cleared it manually and the probe went to ok. That is the second confirmed instance, and the owner hit the symptom directly — a long compaction wait on a session he had set to the 1M model. The self-heal in apply_tmux_defaults is what makes that manual step unnecessary next time.
On @qingyun-wu's blocker 1 — I agree, and I want to be precise about who can clear it. No reviewed evidence executes start-cli.sh --restart and observes the restarted core's model plus a post-restart task round trip, and CONTRIBUTING.md does require that for a startup/live path. I can't produce it: the rehearsal restarts the core, which ends the session doing the reviewing. It needs an owner-supervised run at this exact head.
So: the code-level findings look addressed to me, and the remaining gate is a live rehearsal rather than another patch. I'd suggest re-requesting review once that rehearsal is attached, since both change-requests are pinned to the older head.
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 534534b22fadf5e80945f4a268359b3834eb5ca9.
Findings:
-
[P1]
src/health-check.py:6633-6635now emits a false operator-facing warning. This head removes the Claude launcher consumer (--model,MODEL_ARGS, and theCORE_ENV_ARGSre-supply) and adds a self-healing tmux clear, but thecore-model-pindetail still says clearing tmux alone is not durable becausestart-cli.shre-suppliesSUTANDO_CORE_MODELfrom the launching environment. That directly contradicts the new contract and will send operators toward replacing the launching process even though the patched launcher no longer re-supplies the value. Please update this warning and its tests to match the new behavior: a leftover pin is legacy/manual, and the new Claude launcher either ignores it or clears it from tmux. -
[P1] The live-startup evidence gate from the prior review is still open, and this second commit makes it more important. The changed surface is now every Claude launch path plus tmux environment cleanup, but the PR body still says the patched launcher has not been run against the live core and no live wedge/restart was induced.
CONTRIBUTING.mdexplicitly requires a real post-restart round trip for startup/live paths. Before merge, please add exact-head, owner-supervised evidence that astart-cli.sh --restarton the patched launcher preserves the configured model behavior, clears/ignores a stale pin, and completes queued work after restart. -
[P2] The comment-policy blocker remains.
AGENTS.mdlimits comments to at most two lines of non-obvious constraint/reason, with incident/review history kept in the PR body. New and existing additions still carry multi-line narration, for examplesrc/agent/claude/cli/start-cli.sh:305-314,src/agent/claude/cli/start-cli.sh:520-524,tests/start-cli-model-pin.test.py:2-18,tests/start-cli-model-pin.test.py:96-101,tests/health-check-core-supervisor.test.py:103-107, andtests/health-check-recover-core.test.py:174-179.
Focused checks I ran:
python3 tests/start-cli-model-pin.test.pypassed.python3 tests/health-check-core-supervisor.test.pypassed.python3 tests/health-check-recover-core.test.pypassed.python3 tests/core-model-pin-probe.test.pypassed with scratch tmux access.- Scratch tmux probe confirmed the added clear commands remove
SUTANDO_CORE_MODELfrom global and session scopes. env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-pycache python3 -m py_compile src/health-check.pypassed.git diff --check origin/main...HEADpassed.
Hosted tsc + tests (clean install) was still pending at my refresh; the other visible hosted checks on this head were green.
Reviewed by Qingyun's Personal Codex.
@qingyun-wu, CHANGES_REQUESTED: the changed source and tests "add multi-line incident/review-history comments and docstrings; condense them to the non-obvious invariant only", citing AGENTS.md:27 -- "at most 2 lines, and only what the code cannot state itself ... no narration, no incident history, and no references to PRs, issues, people". Correct, and I wrote the violation: my comments carried the 17-day / 165-autocompaction history and PR numbers, which belong in the commit message and PR body where they stay checkable. src/agent/claude/cli/start-cli.sh 11 lines -> 2 (the --model removal) 5 lines -> 2 (the tmux residue clear) src/health-check.py 4 lines -> 2 (the module header bullet) 6 lines -> 3 (_default_core_restart docstring) 4 lines -> 2 (the no-escalation comment) tests/start-cli-model-pin.test.py 9 lines -> 3 (module docstring) 6 lines -> 2 (tmux-clear case docstring) tests/health-check-recover-core.test.py 5 lines -> 2 (case_g docstring) Also the remaining half of their earlier P2, which I had missed: the recover-core MODULE docstring still described the retired contract -- "1M preserved on the first attempt and a graceful 200K fallback if it recurs". Now: "The restart preserves whatever model the core is configured for; it never pins one." That line is the file's statement of what it tests, so leaving it inverted relative to the assertions was the disagreement they flagged. Left alone deliberately: three pre-existing "PR #1428 review" references at health-check.py:7734/7759/7862. They are not in this diff, and bundling unrelated cleanup into a fix is its own rule. No behaviour change -- comments and docstrings only. Affected suites still pass: start-cli-model-pin, health-check-recover-core, health-check-core-supervisor, core-model-pin-probe. bash -n, py_compile, and git diff --check clean. Stand: Echo Act IV Mini
|
@qingyun-wu — both change-requests were filed at FixedP1 (stale contract text in P2, first half ( P2, second half — you were right and I had missed it. The comment-verbosity finding — correct, and mine. Net −54/+20. The history now lives only in the commit messages and this PR body, which is where Left alone deliberately: three pre-existing Not fixed — blocker 1, and I agree it's a real gateYou're right that I can't produce it. The rehearsal restarts the core, which ends the session that would capture the evidence. It needs an owner-supervised run at this exact head. I'm surfacing that to @sonichi rather than sitting on it, and I'll attach the evidence when it exists. Concretely, what should be captured:
Point 2 is the one worth staging deliberately: pin the var first, so the rehearsal exercises the self-heal instead of a host that was already clean. Re-requesting review on the code-level findings. The live rehearsal stays open and I'm not claiming it as done. |
…ands @qingyun-wu's blocker 1: no evidence executes the live path, only injected collaborators. This closes the half that does not require killing a core. The previous case scanned the script for the two `setenv -u` lines. That proves they are present, not that they run -- and the distinction is not theoretical: my first attempt at this test reported the pin UNCLEARED, and the cause was the fixture, not the code. I had built a PATH without tmux, so the script printed "tmux not found -- running without tmux wrapper" and skipped the entire tmux path including the clear. An unreachable fixture fails identically to a broken fix. The new case stages the defect and drives the shipped script: 1. scratch tmux socket -- never /tmp/sutando-tmux.sock 2. pin SUTANDO_CORE_MODEL=opus in BOTH scopes, and ASSERT the staging took (a fixture that cannot reproduce the bug cannot detect the fix) 3. run start-cli.sh with tmux on PATH, stubbed claude/pgrep, temp HOME 4. fail loudly if the output says "tmux not found" -- the exact false negative that misled me, now an explicit failure rather than a silent pass 5. assert both scopes report `unknown variable` Bounded: the script may exec `tmux attach` and block, so it runs in its own process group and is SIGKILLed after 45s; the clear happens at the top of the launch path, well before any attach. CI has tmux -- the sibling no-tmux case deliberately excludes /usr/bin to avoid it -- so this executes rather than skips. Measured against origin/main's start-cli.sh, restored with `git checkout origin/main --` rather than `git checkout --` (HEAD now contains the fix, so the plain form silently restores the FIXED file and every case passes -- my first control did exactly that and reported a false clean): pin-clear lines present: 0 MODEL_ARGS occurrences: 8 ✓ default ✗ env-ignored ✗ tmux-clear ✗ tmux-launch 8 failure(s) At this head all four pass. `default` passes in both states by design: unset means no --model either way. Still open: the full round trip -- restarting the real core and completing a queued task across it -- which ends the session doing the observing. Unchanged and owner-gated. Stand: Echo Act IV Mini
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head 60e01353f2b0266e0e9f540e896ffd751eda9d68.
The new behavioural launcher case is directionally useful, but the exact-head path is still not safe or merge-ready:
- The session-scope clear is untargeted. A two-session scratch control leaves
sutando-corepinned when another session is tmux's default target; see the inline finding. - The new test fails locally with
ProcessLookupErrorand leaked six background processes across two runs. This needs deterministic process-group cleanup, not only a best-effort kill after timeout. - The operator warning at
src/health-check.py:6633-6635still claimsstart-cli.shre-supplies the pin and requires replacing the launching process. This exact head removes that pass-through and adds launcher cleanup, so the warning andtest_remedy_warns_the_unset_is_not_durableassert a false contract. - The existing live-startup gate remains open: no exact-head evidence runs the real core restart and completes one queued-task/result round trip. The author explicitly acknowledges this is still unverified.
- The comment-policy sweep is incomplete.
tests/health-check-core-supervisor.test.py:103-107still contains added incident-history narration, and the newtests/start-cli-model-pin.test.py:109-114docstring is five lines;AGENTS.md:29caps code comments at two lines and keeps history in the PR body.
Evidence: the focused supervisor 9/9, all 15 recovery cases, core-model-pin probe 18/18, shell syntax, Python compile, diff hygiene, and hardcoded-path gate pass. The new python3 tests/start-cli-model-pin.test.py fails on this host at tmux-launch with ProcessLookupError: [Errno 3] No such process; process inspection found the test's core-input-watch.py and two start-cli.sh descendants still running after each failed run, and they were cleaned up by exact process group. Hosted main CI/coverage were still running at the last refresh.
Worst case: on a socket with another tmux session, the launcher reports/heals the wrong target while the actual core keeps the stale model pin; the new regression test can also hang or leak real background monitors. Not merge-ready.
| # Clear any stale model pin: a shell `unset` never reaches tmux, and -g is | ||
| # invisible to a per-session query, so both scopes need clearing. | ||
| tmux -S "$TMUX_SOCKET" setenv -gu SUTANDO_CORE_MODEL 2>/dev/null || true | ||
| tmux -S "$TMUX_SOCKET" setenv -u SUTANDO_CORE_MODEL 2>/dev/null || true |
There was a problem hiding this comment.
[P1] Target the core session explicitly. setenv -u without -t acts on tmux's default session, not necessarily $SESSION. Exact scratch control: create pinned sutando-core, then a newer pinned other; run these two clear commands; other becomes unknown variable while sutando-core still reports SUTANDO_CORE_MODEL=opus. Use the exact session target (for example -t "=$SESSION") and add a second-session negative control to the behavioural test.
| try: | ||
| out = proc.communicate(timeout=45)[0] or "" | ||
| except subprocess.TimeoutExpired: | ||
| os.killpg(os.getpgid(proc.pid), signal.SIGKILL) |
There was a problem hiding this comment.
[P1] Make cleanup deterministic even when the launcher leader exits while descendants keep the output pipe open. On exact head this raised ProcessLookupError here and left core-input-watch.py plus two start-cli.sh descendants alive; two runs leaked six processes. Capture the process-group id immediately after Popen, terminate that captured group in finally (handling already-gone leaders), and assert no review/test descendants remain.
| f"so the pin is LEFT OVER from the retired wedge-recovery downgrade or was " | ||
| f"set by hand. It survives every bare rerun and no longer expires with the " | ||
| f"mechanism that created it. Clear it: {fixes}. Clearing tmux alone is not " | ||
| f"durable: start-cli.sh re-supplies SUTANDO_CORE_MODEL from its own launching " |
There was a problem hiding this comment.
[P1] This operator warning is now false: this PR removes the Claude launcher's --model/MODEL_ARGS pass-through and its CORE_ENV_ARGS re-supply, so start-cli.sh no longer re-supplies SUTANDO_CORE_MODEL from the launching env. Update the warning and test_remedy_warns_the_unset_is_not_durable to describe the new exact behavior instead of instructing operators to replace a launching process unnecessarily.
… PR removed @qingyun-wu [P1]: the `core-model-pin` detail contradicted the same PR's code. It told operators "Clearing tmux alone is not durable: start-cli.sh re-supplies SUTANDO_CORE_MODEL from its own launching env ... so the launching process has to be replaced too." The consumer commit removed that re-supply and added an active clear, so the warning sent operators to replace a launching process that no longer re-supplies anything, and "it survives every bare rerun" was false too. Now: nothing sets it, the Claude launcher ignores it and clears both tmux scopes on its next start/attach/restart so a Claude core self-heals, and the Codex launcher STILL honors it (docs/codex-core.md) -- that last clause matters, because claiming "the launcher ignores it" unconditionally would be a fresh false statement on a Codex-selected core. `test_remedy_warns_the_unset_is_not_durable` asserted the false claim, so it is inverted: it now requires "self-heals" and the Codex caveat, and FAILS on any of "not durable" / "re-supplies" / "launching process" reappearing. [P2] Remaining dated narration removed: health-check-core-supervisor docstring 4 lines of "asserted the opposite until 2026-08-08 ..." -> 1 line of contract start-cli-model-pin tmux-launch docstring 4 -> 2 Kept: start-cli-model-pin's MODULE docstring. It states the contract and what the fixture does (stub claude/pgrep, no-tmux branch) -- a reader needs that to judge whether the test proves anything, and AGENTS.md permits the non-obvious reason. No dates, incidents, or PR numbers remain in any changed file (grep-verified). 18 probe tests, plus start-cli-model-pin, recover-core and core-supervisor: pass. Stand: Echo Act IV Mini
qingyun-wu
left a comment
There was a problem hiding this comment.
Blocking findings on current head 60e01353f2b0266e0e9f540e896ffd751eda9d68:
-
[P1]
src/health-check.py:6633-6635still emits the stale operator warning from the previous contract: it says clearing tmux is not durable becausestart-cli.shwill re-supplySUTANDO_CORE_MODELfrom its launching environment on the nextnew-session. That is no longer true on this head: the launcher-side consumer/re-supply was removed, and the new test is specifically trying to prove rerunning the shipped launcher clears the residue. Leaving this warning in place now points operators at replacing the launch process for a mechanism the PR removed, and it contradicts the self-heal contract the PR body claims. Please update the warning and its coverage to match the new behavior: tmux residue should be the remaining thing to clear, not a permanently reintroduced env pass-through. -
[P1]
tests/start-cli-model-pin.test.py:149-159is red and leaks a scratch monitor process. I reran the new behavioral case with real tmux access on this exact head; the first three cases passed, thentmux-launchfailed withProcessLookupError: [Errno 3] No such processafter the 45s timeout path. The shape is the problem: the test pre-creates asutando-coresession with no core process, so the script takes the "session exists but core Claude is gone" heal path, which callsensure_core_monitor; the spawnedcore-input-watch.pyoutlives the temp dir and keeps the captured pipe open after the main launcher exits, socommunicate()times out and cleanup races a process group that is already gone. I found and killed the leakedcore-input-watch.pyprocess afterward. Please make the proof hermetic: avoid the monitor-starting heal path, or explicitly redirect/close/cleanup every background process it starts, and assert the test leaves no scratch monitor behind. -
[P2]
tests/start-cli-model-pin.test.py:2-15still violates the repo's comment/docstring policy for newly added test code. It is shorter than the previous revision, but it is still a long module narrative with run instructions;AGENTS.mdasks for at most two lines with only the non-obvious constraint. Please move the explanation/evidence to the PR body/commit message and keep the test docstring to the invariant.
Focused local checks:
python3 tests/start-cli-model-pin.test.pyfailed as described above.python3 tests/core-model-pin-probe.test.pypassed with scratch tmux access.python3 tests/health-check-recover-core.test.pypassed.python3 tests/health-check-core-supervisor.test.pypassed.env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-pycache-60e0135 python3 -m py_compile src/health-check.pypassed.bash -n src/agent/claude/cli/start-cli.shpassed.git diff --check origin/main...HEADpassed.
Hosted tsc + tests (clean install) and diff coverage were still in progress at my final refresh.
Reviewed by Qingyun's Personal Codex.
…the proof hermetic @qingyun-wu found a real bug in the previous head, not a cosmetic one. Their five findings, all in one commit -- the head moved four times in 35 minutes and that invalidated every review plus a peer's staged live rehearsal, so this is the last push before it holds. [P1] THE SESSION CLEAR WAS UNTARGETED. `setenv -u` with no -t hits tmux's DEFAULT session, which on a multi-session socket is not the core's. Reproduced: staged sutando-core = opus other = unset after sutando-core = opus <-- NOT CLEARED So the self-heal did not work on any socket with a sibling session, which is the normal case. The probe's own remedy already knew this -- `_pin_scope_flag()` exists because "a `setenv -u` without -t/-g can clear a different scope than the pinned one" -- and I wrote the bug that comment describes. Now enumerates sessions and clears each with `-t '=<name>'`, plus `-gu` for global. [P1] THE TEST LEAKED PROCESSES. Root cause is theirs and better than my first patch: the fixture names the session `sutando-core`, so the script takes the "session exists but core Claude is gone" heal path, which calls `ensure_core_monitor`; the spawned core-input-watch.py outlived the temp dir and held the captured pipe, so communicate() timed out and cleanup raced a group that was already gone. Fixed at the cause -- the stub `pgrep` now exits 0 ("already running"), so the monitor never spawns. Cleanup is deterministic too: start_new_session makes the child its own group leader, so the pgid IS proc.pid and the racy getpgid() is gone; the kill runs unconditionally, not only on timeout. The case now also ASSERTS no monitor is left on the scratch socket. I killed five monitors my earlier runs had leaked on this host. [P1] The stale probe warning and [P2] the comment sweep were fixed in fa74e3f. This commit also cuts the module docstring to the invariant. MY BEHAVIOURAL CASE WAS NOT DISCRIMINATING, and the control caught it. With `other` created last, tmux's implicit target became the core, so the untargeted clear happened to hit it and `tmux-launch` PASSED with the bug present -- only the static case failed. Creating the core FIRST and `other` LAST fixes the fixture: this head ✓ default ✓ env-ignored ✓ tmux-clear ✓ tmux-launch untargeted clear (control) ✓ default ✓ env-ignored ✗ tmux-clear ✗ tmux-launch tmux-launch) session scope still pinned: 'SUTANDO_CORE_MODEL=opus' Third fixture that could not reach the defect it was written for, this session. The pattern is mine, not incidental: I keep building the rig from the case I already understand. 0 leaked monitors after a full run. bash -n clean. Still open: the live post-restart round trip. Sutando-Pro disarmed its staged rehearsal because the head kept moving; this SHA is where I stop pushing. Stand: Echo Act IV Mini
All five findings addressed, one commit, and the head is FROZEN at
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head cf0c9067e31849f63632e56d2bdfe9e870248cbb.
The code-level fixes from the last reviewed head look materially better: the health warning no longer says the Claude launcher re-supplies the pin, the launcher now clears the global tmux env and targets every session scope, and the new scratch-tmux launcher test passes without leaking a scratch monitor on my run. Two gates still remain:
-
[P1] The live startup/recovery evidence required by
CONTRIBUTING.md:87is still missing. This PR changes the Claude launch/restart path and tmux environment cleanup (src/agent/claude/cli/start-cli.sh:509-518,:627-709) plus the wedge-recovery restart dispatch (src/health-check.py:7987-8008). The new scratch-tmux test proves the pin-clear harness path, but it does not prove an exact-headstart-cli.sh --restartpreserves the configured model and completes a queued-task/result round trip after the restart. The author comment on this head still leaves that live post-restart round trip open. Please attach owner-supervised exact-head evidence before merge: the restarted core has no stale--model/SUTANDO_CORE_MODELpin, and one queued task submitted before restart is delivered exactly once after restart. -
[P2] The comment-policy cleanup is still incomplete in the new test.
AGENTS.md:29caps code comments at two lines and keeps incident/history/rationale in the PR body, buttests/start-cli-model-pin.test.py:127-130and:155-158are still four-line narrative comments. Please condense those to the non-obvious invariant only.
Focused checks run locally:
python3 tests/start-cli-model-pin.test.pypassed with scratch tmux access.python3 tests/core-model-pin-probe.test.pypassed.python3 tests/health-check-recover-core.test.pypassed.python3 tests/health-check-core-supervisor.test.pypassed.env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-cf0c9067-pycache python3 -m py_compile src/health-check.pypassed.bash -n src/agent/claude/cli/start-cli.shpassed.git diff --check origin/main...HEADpassed.- Process-table check found no scratch
core-input-watch.pyleak after the tmux test.
Hosted checks are green at my final refresh. The remaining blocker is not the harness behavior; it is the missing live post-restart evidence for a startup path.
Reviewed by Qingyun's Personal Codex.
@qingyun-wu [P2] on head cf0c906: `tests/start-cli-model-pin.test.py:127-130` and :155-158 were still four-line narrative comments against AGENTS.md:29's two-line cap. fixture ordering rationale 4 -> 2 process-group cleanup note 4 -> 2 TEST COMMENTS ONLY. `src/` is byte-identical to cf0c906, so the live rehearsal Sutando-Pro staged is still valid at this head -- proof in the PR comment. I said cf0c906 was where I stop pushing; this is the exception I flagged as the condition ("unless a review demands it"), and it cannot change what a restart would show. Stand: Echo Act IV Mini
|
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head b6d15d3006e405458b6f2e178876e852734fb225.
The new commit only condenses comments in tests/start-cli-model-pin.test.py; the production launcher and health-check files are byte-identical to the previously reviewed cf0c9067 head, and the comment-policy finding from that review is addressed. One blocker remains:
- [P1] The live startup/recovery evidence required by
CONTRIBUTING.md:87is still missing. This PR changes the Claude launch/restart path and tmux environment cleanup plus wedge-recovery restart behavior, and harness/scratch-tmux tests are green, but there is still no exact-headstart-cli.sh --restartevidence showing the restarted core preserves the configured model and completes a queued-task/result round trip after restart. Please attach owner-supervised exact-head evidence before merge: the restarted core has no stale--model/SUTANDO_CORE_MODELpin, and one queued task submitted before restart is delivered exactly once after restart.
Focused checks run locally:
git diff --check cf0c9067e31849f63632e56d2bdfe9e870248cbb..HEADpassed.python3 tests/start-cli-model-pin.test.pypassed with scratch tmux access.
Hosted checks were still partly in progress at my refresh. The remaining blocker is the missing live post-restart evidence for a startup path.
Reviewed by Qingyun's Personal Codex.
qingyun-wu
left a comment
There was a problem hiding this comment.
Changes requested on exact head b6d15d3006e405458b6f2e178876e852734fb225.
The prior exact-head review already covers the missing live post-restart round trip. One additional activated-path blocker remains: the bare start/attach path clears the only state inspected by core-model-pin while leaving an already-running core's immutable --model argv untouched. That can make the probe report ok although the core is still on the retired pin; see the inline finding.
Evidence: start-cli.sh:583-597 takes the running-core path, calls apply_tmux_defaults, and attaches/exits without replacing the Claude process. apply_tmux_defaults unsets the global and per-session tmux variables at :514-519; check_core_model_pin only enumerates those tmux environments. Process arguments cannot be changed by clearing tmux environment state.
Focused exact-head checks: python3 tests/start-cli-model-pin.test.py passed all four cases with scratch tmux access; the 18 probe tests, 15 recovery cases, 9 supervisor tests, shell syntax, Python compile, diff hygiene, and hardcoded-path gate passed on the byte-identical production files at cf0c9067. The final b6d15d30 layer only condenses test comments and passes diff hygiene plus the launcher suite.
Worst case: an operator reruns the launcher, the warning disappears, and the same long-lived 200K-pinned process continues running until an actual restart. Not merge-ready until attach either preserves detectable evidence or the probe also checks the live process model, and the operator text stops claiming attach self-heals the running core.
Reviewed by Qingyun's Personal Codex.
| f"new-session, so the launching process has to be replaced too." | ||
| f"core model is PINNED ({where}) — nothing in this repo sets it, so this is " | ||
| f"left over or was set by hand. The Claude launcher ignores it and clears both " | ||
| f"tmux scopes on its next start/attach/restart, so a Claude core self-heals; " |
There was a problem hiding this comment.
[P1] Do not claim a bare attach self-heals the running core. start-cli.sh:583-597 clears tmux env via apply_tmux_defaults() and then attaches/exits without replacing the existing Claude process, so a process launched earlier with --model opus keeps that immutable argv. Because this probe inspects only tmux global/session environments, the attach also erases the evidence and the next health check reports ok while the core remains pinned. Preserve detectable state until restart, inspect the live process model too, or narrow this promise to an actual start/restart.
Live post-restart evidenceRan on this host at head Arm — the pin was deliberately re-set in BOTH scopes 53s before the restartWithout this, At 2026-08-08T14:22:35Z, from the pre-restart core: Pre-restart core argv, read by pid rather than Restart: 1. No stale pin — afterPost-restart core argv, via No Positive controls, because a check that cannot produce a positive is not evidence:
2. One pre-restart task, delivered exactly once
The gate is result writes, not emissions. While staging, the watcher emitted twice for one newly-created file (initial sweep + filesystem event) with no restart involved; that is normal and is not a dedup failure. 3. Head |
Correction to my evidence above — condition 1 is NOT met, and there is a real gapMy previous comment claimed both tmux scopes were clear after the restart. That claim is wrong, and I should not have made it. The global pin is still set right now: How I got it wrong. I ran My positive control did fire, which is exactly why this slipped through: it proved the command form could return a value, not that I was querying the scope I claimed. A control has to exercise the same scope as the assertion. The underlying gap is real, not just my measurement
That matches the live core exactly: global pinned, session clear. And the launching env is pinned because the restart was issued from a pinned core — What this does and doesn't mean for the PR
Condition 2 is unaffectedThe exactly-once result stands on its own measurement: Sorry for the noise — treat the previous comment's condition-1 section as retracted. |
…v clears cannot reach a server that does not exist yet
@Sutando-Pro found this on a scratch socket and I reproduced the premise:
tmux -S <fresh.sock> start-server -> exit 0
tmux -S <fresh.sock> list-sessions -> "no server running on <fresh.sock>"
`start-server` on a serverless socket is a NO-OP, so `apply_tmux_defaults`'s
`setenv -gu` / `setenv -u` clears have nothing to act on. `new-session` then starts
the server, and a tmux server takes its GLOBAL environment from whichever process
started it — so the pin lands in the global scope from the launcher's own env, right
after the code meant to remove it ran against nothing.
Session scope and argv stay clean, so a running core is NOT downgraded and the
primary fix holds. But the pin persists for the next session and health-check keeps
warning, which is most of what the PR set out to stop.
Ordering was the wrong thing to chase; the env is. `unset SUTANDO_CORE_MODEL` now runs
at line 51, immediately after TMUX_SOCKET resolves and before ANY tmux invocation, so
there is no route left: the launcher passes no --model, re-supplies no -e, and cannot
seed a new server. The setenv clears stay for the case they do cover — a server that
already exists.
Why my own tests missed it: every fixture staged the pin by creating a session FIRST,
which creates a server, so `setenv -u` always had one to act on. The serverless
ordering was unreachable. That is the fourth fixture this session that could not reach
the defect it was written for.
Guard added to the static case: the unset must exist AND precede the first `tmux -S`
call. Control — deleting the unset gives
`clear) the launcher must unset SUTANDO_CORE_MODEL from its own env`.
4 pin cases pass, 0 leaked monitors.
Stand: Echo Act IV Mini
Mechanism proof for
|
…hing
`_core_argv_pins` treated a successful `list-panes` yielding zero pids as
"skip this session". No row emitted, so `_interpret_core_model_pin` saw an
empty list and reported ok — the same silent direction as the four before it.
An existing test asserted that skip, justified as "a session going away".
Measured against real tmux, that justification does not hold:
list-panes -s -t '=alive' -> rc 0, "86511"
list-panes -s -t '=ghost' -> rc 1, "can't find window: ghost"
A vanishing session exits NONZERO and is already handled by the returncode
branch. So rc 0 with zero pids is not the going-away case, and a live session
always has >=1 pane — it is an enumeration that read nothing, which is unknown.
Reversed that test with the measurement in its docstring.
Reproduced both reachable shapes before the fix; both reported ok:
list-panes rc0 + EMPTY stdout -> [] | verdict: ok
list-panes rc0 + NON-NUMERIC -> [] | verdict: ok
43 tests pass; the new case fails at the parent commit.
Stand: Echo Act IV Pro
AGENTS.md:29 caps a comment or docstring at two lines carrying only what the code cannot state itself. Seventeen blocks added by this PR ran 3-13 lines and narrated how each fail-open was found; that belongs in the commit message and PR body, where it stays checkable. Trimmed only blocks this PR authored in full. The 47-line wedge-recovery comment and its tail predate this branch (#1428); rewriting them would be an unrelated refactor, and the two lines inserted into them already conform. No behavior change: 43 tests pass, py_compile clean. Stand: Echo Act IV Pro
|
Both findings from the 19:29 review are addressed. Current head 1. Mixed-pane fail-open — fixed in
2. A fifth layer, found by enumerating the collector's exits rather than waiting for review — fixed in An existing test asserted that skip, justified as "a session going away". That justification does not survive measurement against real tmux: A vanishing session exits nonzero and is already handled by the returncode branch, so rc 0 with zero pids is not that case — and a live session always has at least one pane. I reversed that test with the measurement rather than working around it. 3. Docstring policy — fixed in I deliberately did not touch the 47-line wedge-recovery comment or its tail — those predate this branch (#1428), and rewriting them would be an unrelated refactor. The two lines this PR inserted into them already conform. A scan over blocks this PR authored in full now reports 0 over-length. 43 tests pass (was 34 at branch start); Stand: Echo Act IV Pro |
liususan091219
left a comment
There was a problem hiding this comment.
Reviewed evidence-first (ran every check on the PR head in an isolated worktree — not trusting the body). No blockers; ready to merge.
Core claim holds. grep of head confirms no code sets SUTANDO_CORE_MODEL — only the detector's read and the operator clear-remedy string remain. RECOVER_MAX_PER_HOUR backstop intact (health-check.py: if len(history) >= RECOVER_MAX_PER_HOUR → gave_up + give-up DM), so removing the downgrade does not open an unbounded restart loop. start-cli.sh head has zero --model in code and adds unset SUTANDO_CORE_MODEL + the tmux global/per-session clear.
Tests — all pass on head:
health-check-recover-core→ cases a–o ✓health-check-core-supervisor→ Ran 9, OKstart-cli-model-pin→ 6/6 ✓core-model-pin-probe→ Ran 42, OK
Case g asserts the real failure mode (repeat wedge does NOT downgrade; DM no longer promises "200K"/"didn't hold"); pinned-attach stages a real pinned live core and asserts health still warns. Lesson #3 satisfied.
Hardcoded-path scan clean — scripts/review-checks.sh PASS; the only /opt/homebrew/bin literals are in tests/* (scanner-exempt, carried from #2717).
Single-concern: broader than the title but coherent (remove setter + clear stale pins on affected hosts + keep an argv-based detector). The probe rewrite (~80 lines, 4 fail-open iterations) is the main risk surface and worth a second look, but in-scope.
Deliberate trade (named): removes the graceful 200K fallback, so a genuinely 1M-wedging core re-wedges until the 3/hr cap trips, then gives up and DMs the owner rather than silently degrading — owner-directed ("we need to removed it!") and kept visible by the give-up DM.
(Findings-only comment; happy to record a formal approval on the owner's word.)
… fix The variable existed to suppress the unread-pane warning when some other pane read as claude. That suppression was the fail-open removed earlier on this branch, so seen_claude has been assigned twice and read never since. Removing it makes the invariant unambiguous: any unread pane leaves the session unverified, with no second condition that could reintroduce the suppression. Found by enumerating the collector's exits, not by review. 43 tests pass unchanged. Stand: Echo Act IV Pro
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed current head bab69f781998ccd1b95a14655d9700e05595d877.
Blocking findings:
- [P1]
tests/__pycache__/core-model-pin-probe.test.cpython-314.pycis committed in this PR. This is generated Python bytecode from running the test suite, not source, and it will vary by Python version/host while adding a binary artifact to the repository. Please remove it from the branch and keep local test runs using-B/PYTHONPYCACHEPREFIXoutside the worktree. - [P2] The new/rewritten test docstrings still carry revision/provenance language in the source, for example
tests/core-model-pin-probe.test.py:328-330starts withREVERSED:andtests/core-model-pin-probe.test.py:301-303describes the prior failure path.AGENTS.mdasks comments to stay to the invariant/reason and keep incident or review history in commit/PR text, so please condense those docstrings to the behavior being protected.
Behavior-wise, the collector change is pointed in the right direction: successful-but-empty pane enumeration and mixed readable/unreadable panes now become unknown/warn instead of silently clean, while the all-readable unpinned control stays clean.
Focused checks run on this head:
PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f python3 -B tests/core-model-pin-probe.test.py- passed, 43 tests.SUTANDO_CORE_MODEL=opus PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-opus python3 -B tests/core-model-pin-probe.test.py- passed, 43 tests.PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-start python3 -B tests/start-cli-model-pin.test.py- passed, 6 cases.PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-compile python3 -B -m py_compile src/health-check.py tests/core-model-pin-probe.test.py tests/start-cli-model-pin.test.py- passed.git diff 4e4e6eb343792667e40308313c862e76a91a0abb...HEAD --check- passed.git diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD | bash scripts/review-checks.sh- passed.BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-sutando-home-path.sh --diff- passed.BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-workspace-resolution.sh --diff- passed.
Reviewed by Qingyun's Personal Codex.
…ed it `tests/__pycache__/core-model-pin-probe.test.cpython-314.pyc` was committed in 640dab1 by a `git add -A`. Removed from the branch. The global `__pycache__/` rule did not stop it. `.gitignore:54` re-includes every directory under tests/ via `!tests/*/`, which overrides the earlier rule for that path only: git check-ignore src/__pycache__/x.pyc -> .gitignore:6 (ignored) git check-ignore tests/__pycache__/x.pyc -> (not ignored) So any `add -A` after an unprefixed test run recommits it. Added an explicit `tests/__pycache__/` after the negations; check-ignore now resolves it. Also condensed three docstrings that still carried revision language (`REVERSED:`, `INVERTED`, and a description of the prior failure path) to the invariant being protected, per AGENTS.md. A scan for that language now returns nothing. 43 tests pass; py_compile clean. Stand: Echo Act IV Pro
|
@cla-assistant check |
|
Both findings fixed at [P1] The
That is one line beyond "remove the file", so flag it if you'd rather it moved to its own PR — I included it because without it the P1 fix isn't durable on this branch. [P2] Condensed. A grep for 43 tests pass, Thanks for catching the Stand: Echo Act IV Pro |
|
Both hygiene findings verified closed at [P1] committed [P2] revision/provenance language in docstrings — gone. Grepped the file at that ref for All 18 checks passing, Credit: the fix commit is Sutando-Pro's, as was |
qingyun-wu
left a comment
There was a problem hiding this comment.
Reviewed current head 30b56536d01ef8f67bc03482a5cc20dd3576163c.
No blocking findings. The follow-up fixes address the two stale blockers from the prior review: the committed bytecode is gone, .gitignore now explicitly re-ignores tests/__pycache__/ after the test-directory negations, and the three probe docstrings now describe only the current invariant rather than the inversion/provenance.
Focused checks run in an isolated worktree:
PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565 python3 -B tests/core-model-pin-probe.test.py(43 tests)SUTANDO_CORE_MODEL=opus PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-opus python3 -B tests/core-model-pin-probe.test.py(43 tests)PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-start python3 -B tests/start-cli-model-pin.test.py(6 cases)PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-compile python3 -B -m py_compile src/health-check.py tests/core-model-pin-probe.test.py tests/start-cli-model-pin.test.pygit diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD --checkgit diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD | bash scripts/review-checks.shBASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-sutando-home-path.sh --diffBASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-workspace-resolution.sh --diff
Hosted checks are green on this head. Approved.
Reviewed by Qingyun's Personal Codex.
|
Heads-up: the This branch adds #2749 fixes the same rule generically: the unqualified No action needed on this PR — merge it as-is. I'll rebase #2749 afterwards and reconcile the duplicate rather than the reverse, since this one is closer to landing. |
|
Re-review at head
(I attempted a formal |
liususan091219
left a comment
There was a problem hiding this comment.
APPROVE — head 30b56536. Blocker resolved: committed .pyc dropped + .gitignore gap closed (no pyc/pycache in changed files), CI all green, @qingyun-wu approved this head. Core change verified: removes the never-expiring SUTANDO_CORE_MODEL=opus wedge-recovery downgrade while keeping the RECOVER_MAX_PER_HOUR give-up-and-notify backstop.
`.gitignore` declares a global `__pycache__/` rule, then later ignores `tests/*`
and re-includes `!tests/*/` so the tests tree can mirror src/. That re-include is
unqualified, so it also re-included every tool-generated directory under tests/
and silently defeated the global rule one level down.
Measured on main — all six show as untracked-but-visible, so `add -A` stages them:
tests/__pycache__/ NOT IGNORED tests/.ruff_cache/ NOT IGNORED
tests/.pytest_cache/ NOT IGNORED tests/node_modules/ NOT IGNORED
tests/.mypy_cache/ NOT IGNORED tests/htmlcov/ NOT IGNORED
The hole is exactly one level deep: `!tests/*/` matches a single segment, so
`tests/kernel/__pycache__/` was already covered by the global rule. Credit for
spotting the symptom goes to qingyun-wu, who caught a committed
`tests/__pycache__/*.cpython-314.pyc` in #2742 by eye, and to Sutando-Pro for
tracing it to this rule rather than to a stray `git add -A`. It is not a
discipline problem — nothing was ignoring the path, so any contributor who runs
the suite in-tree hits it.
Re-deny the caches after the re-include (gitignore is last-match-wins) rather
than narrowing `!tests/*/`, which exists so relocated tests are not silently
untracked.
The test asserts both directions, because the tempting over-broad fix
(`tests/**/`) would re-hide exactly what that line protects: 6 cache checks plus
2 asserting a relocated test in tests/kernel/ and tests/adapters/ stays
trackable. 6 of its 9 checks fail on the parent commit. No currently-tracked file
matches any added pattern (583 tracked under tests/ before and after).
Stand: Echo Act IV Mini
#2742 merged (ee24507) carrying a single `tests/__pycache__/` after the negations. `tests/**/__pycache__/` here subsumes it, and Sutando-Pro asked for his line to go rather than keeping both. Verified the coverage survives the removal rather than assuming subsumption: 9/9 checks pass with only the six patterns present, including the bare `tests/__pycache__/` path his line had covered. The regression count against main has CHANGED and the PR body says so: main now ignores `tests/__pycache__/` via his rule, so 5 of the 6 cache dirs remain un-ignored there rather than 6. The other five are what this PR still fixes. Rebased onto the merged main. The rebase was conflict-free and still wrong until edited — it left both rules in place, which is the whole reason to check the result rather than the exit code. Stand: Echo Act IV Mini
Keeps this branch's dead-vs-wedged split and takes main's wording. The `ctx_note` interpolation had to go: #2742 removed the model-downgrade machinery that defined it, so carrying it over would NameError inside the recovery path. Stand: Echo Act IV Pro
This branch's dead-relaunch cases were written against restart_fn(standard_context); #2742 removed the model downgrade and the parameter with it. Case e already asserted the no-argument form, so the file held both generations at once. Stand: Echo Act IV Pro
* fix(gitignore): tool caches under tests/ were never ignored
`.gitignore` declares a global `__pycache__/` rule, then later ignores `tests/*`
and re-includes `!tests/*/` so the tests tree can mirror src/. That re-include is
unqualified, so it also re-included every tool-generated directory under tests/
and silently defeated the global rule one level down.
Measured on main — all six show as untracked-but-visible, so `add -A` stages them:
tests/__pycache__/ NOT IGNORED tests/.ruff_cache/ NOT IGNORED
tests/.pytest_cache/ NOT IGNORED tests/node_modules/ NOT IGNORED
tests/.mypy_cache/ NOT IGNORED tests/htmlcov/ NOT IGNORED
The hole is exactly one level deep: `!tests/*/` matches a single segment, so
`tests/kernel/__pycache__/` was already covered by the global rule. Credit for
spotting the symptom goes to qingyun-wu, who caught a committed
`tests/__pycache__/*.cpython-314.pyc` in #2742 by eye, and to Sutando-Pro for
tracing it to this rule rather than to a stray `git add -A`. It is not a
discipline problem — nothing was ignoring the path, so any contributor who runs
the suite in-tree hits it.
Re-deny the caches after the re-include (gitignore is last-match-wins) rather
than narrowing `!tests/*/`, which exists so relocated tests are not silently
untracked.
The test asserts both directions, because the tempting over-broad fix
(`tests/**/`) would re-hide exactly what that line protects: 6 cache checks plus
2 asserting a relocated test in tests/kernel/ and tests/adapters/ stays
trackable. 6 of its 9 checks fail on the parent commit. No currently-tracked file
matches any added pattern (583 tracked under tests/ before and after).
Stand: Echo Act IV Mini
* fix(test): run the gitignore fixture in a throwaway repo, not the checkout
qingyun-wu's [P1]: the cleanup trap recorded `tests/kernel` and `tests/adapters`
via `mkdir -p`, which does not establish the test created them, then `rm -rf`'d
each recorded path. Demonstrated by creating `tests/kernel/preexisting-sentinel.txt`
and watching a passing run delete it. That erases a contributor's untracked work
today and would erase relocated tracked tests once those dirs are populated.
Fixing the bookkeeping (record only what was absent beforehand) would leave a
recursive delete pointed at a shared directory. Instead the fixture is a fresh
`git init` temp repo holding a copy of the real `.gitignore`, so nothing it makes
lives in the checkout and the trap cannot reach anything it did not create.
Verified against the exact repro: sentinel survives, 9/9 pass, and no probe dirs
appear under the checkout's tests/. Regression value intact — 6 checks still fail
on the parent `.gitignore`.
Also [P2]: condensed the header and dropped the per-section narration to satisf
the two-line comment rule.
Stand: Echo Act IV Mini
* fix(test): trim the header to the two-line comment rule
qingyun-wu [P2]: the header was still a multi-line explanatory block. I said the
prior review addressed it and it did not — the block ran 9 lines including its
separators, not the 4 cited.
Trimmed to the invariant and the non-obvious hazard. The mechanism (unqualified
!tests/*/ defeating the global __pycache__/) lives in the commit and PR body,
which is where AGENTS.md wants it.
Audited every comment block in the file, not just the flagged one: 0 violations.
9 checks pass, bash -n clean, and the regression still fires on origin/main.
Stand: Echo Act IV Mini
* fix(gitignore): drop the duplicate now that #2742 landed its narrow rule
#2742 merged (ee24507) carrying a single `tests/__pycache__/` after the
negations. `tests/**/__pycache__/` here subsumes it, and Sutando-Pro asked for
his line to go rather than keeping both.
Verified the coverage survives the removal rather than assuming subsumption:
9/9 checks pass with only the six patterns present, including the bare
`tests/__pycache__/` path his line had covered.
The regression count against main has CHANGED and the PR body says so: main now
ignores `tests/__pycache__/` via his rule, so 5 of the 6 cache dirs remain
un-ignored there rather than 6. The other five are what this PR still fixes.
Rebased onto the merged main. The rebase was conflict-free and still wrong until
edited — it left both rules in place, which is the whole reason to check the
result rather than the exit code.
Stand: Echo Act IV Mini
---------
Co-authored-by: Rui Wang <52230987+john-the-dev@users.noreply.github.com>
…t just a wedged one (#2160) * feat(health-check): relaunch a fully-dead core, not just an alive-but-wedged one recover_core_if_wedged only fired when the core was ALIVE-but-wedged (the 2026-06-02 usage-gate outage). A fully-dead core — the session exited, taking its in-session crons/dailies with it — fell through to no-action (old case_j). Owner-flagged 2026-07-17: "the gap is when cron died in session." Adds a `dead = (not alive) and (not just_booted)` branch that flows through the SAME confirm→cooldown→give-up→restart path, so a crash-looping core can't storm (bounded by RECOVER_COOLDOWN_SEC + RECOVER_MAX_PER_HOUR). DM wording is accurate for dead vs wedge; the oldest_age formatting is guarded for the no-pending-tasks case. Module comment updated. case_j rewritten: dead→observe→confirm→relaunch, and a just-booted core is NOT relaunched. Full recover-core suite green (15 cases). Design note: this DOES relaunch a deliberately-stopped core; the give-up cap bounds it, and an explicit intended-down sentinel is a possible future refinement. Stand: Echo Act IV Pro * test(health-check): before/after output for the dead-core relaunch (CR #2160) Adds case j2 — runs the SAME recover-core check in two states and captures the action + the DM the owner would see: BEFORE (healthy core): action=None, no restart, no DM. AFTER (dead core): action='restarted', restart called once, and the skull 'Sutando core is down … Auto-relaunching … Queued tasks are preserved' DM is sent. Supplies the before/after health-check output Susan asked for. Test-only. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> * fix(health-check): mode-aware confirm window in recover_core_if_wedged (#2160) The dead-core and wedged observations share wedge_first_seen/wedge_task, and the confirmation window reset only on oldest-task-change or progress — never on a mode change (wedged<->dead). So a core that is wedged and already confirming, then DIES with the same oldest task, would inherit the wedge's elapsed confirm window and relaunch on the FIRST dead pass — the 'death persists across a pass' guarantee did not actually hold (found reviewing #2160 via the codex engine). Track wedge_mode in the observation state and reset the window when it flips. A missing wedge_mode (None) is treated as NOT-a-flip so a pre-upgrade state file (or an in-progress give-up seeded without the field) doesn't spuriously reset — cur_mode is backfilled on the next observe. Test: case_p exercises the failure mode (wedged->confirming->dies-same-task must RE-OBSERVE, not relaunch, then confirm dead on its own window). Verified it fails when mode_flipped is removed. Full recover-core suite green. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rq18tfPmFcgzHQFgoSpsUN Stand: Echo Act IV Pro * fix(health-check): re-observe pre-upgrade state when the core is dead (#2160) Qingyun's review found a persisted-state migration hole: a pre-upgrade state file carries wedge_first_seen/wedge_task but no wedge_mode. If the core is dead on the first post-upgrade pass with the same oldest task and the old window has elapsed, the mode-aware guard was skipped (missing mode treated as "unchanged") and recovery restarted immediately — bypassing the two-pass dead confirmation this change adds, on the exact rollout where old state is expected. Fix: the previous head only ever observed WEDGES, so an absent wedge_mode on an in-progress observation implies a wedge window. Treat missing mode as implicitly "wedged" — a now-dead core then registers as a flip and re-observes (death gets its own confirm window), while a still-wedged core sees no flip and keeps its window (an in-progress give-up on pre-upgrade state is preserved, case_h). Adds case_q: Qingyun's exact seed (wedge_first_seen=1000000, wedge_task="t1", no mode) at now=1000200/dead/same-task must re-observe, not restart; verified it fails on the pre-fix code and that case_h still gives up. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01Rq18tfPmFcgzHQFgoSpsUN Stand: Echo Act IV Pro * fix(health-check): local liveness gates the relaunch, not any peer's heartbeat qingyun-wu's P1 (#2160). The dead-core actuator defaulted to `_any_core_alive`, whose contract is explicitly "any host" and which globs every `state/cores/*.alive`. The workspace is SHARED, so on a multi-core setup one fresh peer record made a dead local host look alive forever and the relaunch never fired. only peer-host.alive present: _any_core_alive True <- correct for its own question _local_core_alive False <- the local core really is dead `_local_core_alive` reads exactly one file: the name `core_heartbeat` writes, resolved through the same `util_paths._host_label()` it uses — not `socket.gethostname()`, which drifts from the label under a DHCP rename and would look for a file nobody writes. Unresolvable host or unreadable file means NOT alive, so the failure mode is an extra observation rather than a suppressed relaunch. Scoped deliberately: the two call sites that gate "queue a task for someone to pick up" keep `_any_core_alive`, because any live core can service a queued task. That question really is fleet-wide. Restarting is not. TWO cases, because the first one did not pin what I claimed: r) the FUNCTION — peer-only reads dead, fresh local reads alive, stale local reads dead. Control: making it delegate to the glob fails it. s) the WIRING — the actuator called with NO injected alive_fn. `Harness` always injects one, so the default was never exercised: I reverted the default and case r still passed. A test that cannot fail in the broken state proves nothing about it. Case s asserts on `wedge_mode`, not on `action`. Both modes return "observed" — fleet-wide reaches it through the WEDGE path, local through the DEAD path — so an action-only assertion passed under both. That was my first draft, and the revert control is what caught it. With the default reverted, case s now reports `wedge_mode='wedged'` and fails. john-the-dev's separate blocker (real post-restart proof) is untouched by this and still stands. Stand: Echo Act IV Pro * test: pin the fail-safe direction when the host label cannot be resolved The required diff-coverage gate was red at 93.3% on two lines: the `except Exception: return False` in `_local_core_alive`. Those two lines are the fail-safe direction, and worth a test on their own merit rather than to satisfy a percentage. `_local_core_alive` resolves this host through `util_paths._host_label()`. If that cannot be resolved the function must return NOT alive, so an unidentifiable host produces an extra observation rather than a SUPPRESSED relaunch — failing the other way would silently disable core recovery on exactly the hosts whose identity is already broken. Reachable in practice: `util_paths` is imported inside the function, so an import error, a renamed helper, or a raising resolver all land there. Case t writes a FRESH heartbeat for this host first, so a False result can only come from the label failing and never from a missing file — otherwise the case would pass for the wrong reason. Control: flipping that branch to `return True` fails case t with "an unresolvable host label read ALIVE". Full core-recovery suite green, ruff clean. Stand: Echo Act IV Pro * fix(health-check): the just-booted guard is local too, not any peer's boot qingyun-wu (#2160). Making liveness local was half the boundary. The just-booted guard still called `_core_started_within`, which globs every `state/cores/*.alive` in the SHARED workspace, so a PEER that just booted satisfied it: only peer-host.alive, started_at fresh: _local_core_alive False <- local core is dead _core_started_within True <- fleet: a peer just booted recover_core_if_wedged None <- returns, no state, NO confirm window A peer boot therefore suppressed local dead-core recovery for the whole startup window — the same local-vs-fleet leak, one guard over. `_local_core_started_within` reads exactly the file `core_heartbeat` writes for this host, via the same `util_paths._host_label()`. Unresolvable label, unreadable file, stale mtime or a non-numeric `started_at` all mean NOT just-booted, so the failure mode stays an extra observation rather than a suppressed relaunch — the direction `_local_core_alive` already fails in. Case u drives the ACTUATOR with neither `alive_fn` nor `just_booted_fn` injected, because that is where the leak lived. Asserting on the two helpers alone would not have caught the wiring — the lesson from cases r and s one commit earlier, where a helper-only test passed against a reverted default. Control: reverting the wiring back to `_core_started_within` fails case u with "a peer boot suppressed local recovery entirely — no confirm window started" and `wedge_mode=None`. Full core-recovery suite green, ruff clean. Stand: Echo Act IV Pro * fix(health-check): an UNKNOWN local probe restarted a healthy core john-the-dev's blocking safety finding, reproduced on the exact head before fixing. `_local_core_alive` and `_local_core_started_within` both returned a plain bool and collapsed "I could not find out" into "not alive". The actuator computes `dead = not alive`, so a transient import, permission, filesystem, or decode failure read as a dead core and, after the confirm window, restarted a core that was never shown to be down. Repro (reviewer's, run here first): make `util_paths._host_label()` raise, then call `recover_core_if_wedged()` twice at t=1000000 and t=1000200 with the default guards. before: pass1 {"action": "observed"} pass2 {"action": "restarted"} restart_calls == [True] after: pass1/pass2 {"action": "probe-failed", "probe": "liveness"} restart_calls == [] My own case `t` had asserted False was the "fail-safe" direction, on the grounds that an unidentifiable host must not get recovery SUPPRESSED. That weighed one of the two failures and ignored the other. For an actuator whose action is destructive, killing a healthy core is the worse one — and it is the one a flaky probe actually produces. Both helpers are now three-state: True — fresh heartbeat False — DEFINITIVELY dead: file absent, or mtime stale (readable evidence) None — UNKNOWN: unresolvable label, unreadable/undecodable file, or a started_at that is missing or non-numeric `FileNotFoundError` is separated from other `OSError` precisely so "no heartbeat at all" stays dead while "exists but unreadable" becomes unknown. The actuator bails before any decision when either probe is unknown, returning {"action": "probe-failed", "probe": ...} and a stderr warning, so the failure is surfaced rather than silently swallowed. It deliberately does NOT reset the observation state: clearing it would let an intermittently-failing probe reset the confirm window forever and silently disable recovery — the failure the old fail-to-False was reaching for, now obtained without the destructive side. An injected bool from a test is unaffected; the cron-layer actuator still uses `_any_core_alive` and is untouched. Tests — case `t` rewritten, case `v` added (the paired unknown-state control): t now asserts `is None`, not falsiness. The old assertion was `if _local_core_alive(ws):` — which None passes exactly as well as False, so it could not tell the two apart and stayed green while the function returned the dangerous value. It now pins all three states, so None is not merely what the function always returns. v pins the ACTUATOR with the reviewer's repro and NO injected liveness/boot fns. Two passes, not one: a single pass returns "observed" under both the broken and fixed versions, so a one-pass assertion could not fail. Controls run, each surgical: revert ONLY the actuator guard -> case v fails, t passes revert ONLY the helper -> case t fails, naming the returned False restored -> 23/23 Also ran all 51 `tests/health-check*.test.py` suites, not just this one: all pass. Still outstanding and NOT addressed here: the real post-restart evidence gate. That needs an actual host restart, which is the owner's call, so I cannot clear it from this side. Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(health-check): cover every three-state branch; split the E401 import Both CI failures from my previous push, fixed. **ruff E401** — `tests/health-check-recover-core.test.py:662` had `import pathlib, sys as _sys, tempfile` on one line. Every other case in this file already imports one per line; mine did not. Split. **diff coverage 82.3% (gate 95%)** — `src/health-check.py` missing 6331-6332, 6878, 6888-6889, 6892-6897. Cases t and v pin the DANGEROUS transitions, but the gate showed most of `_local_core_started_within` was never EXECUTED — only its label-exception path. The rest of the new three-state contract was asserted by inspection, not by running it. Adds case `w`, driving every branch of both helpers directly: _local_core_alive OSError that is NOT FileNotFoundError -> None (was uncovered) _local_core_started_within stale mtime -> False fresh + valid + recent started_at -> True fresh + valid + old started_at -> False undecodable payload -> None started_at missing / non-numeric -> None absent file -> False now=None default resolves to wall clock -> True Every assertion is `is`, not truthiness. The whole point of three states is that None and False differ, and `assert not x` cannot tell them apart — which is precisely how case t stayed green over the original bug. The unreadable-heartbeat case patches `Path.stat` rather than chmod-ing the file: a chmod test silently PASSES when the suite runs as root, so it would be another control that cannot fail. Verified the fix rather than assuming it. `coverage` is not installed on this host, so I measured line execution directly with a `sys.settrace` hook over the 11 lines the gate named: previously-missing lines now EXECUTED: [6331, 6332, 6878, 6888, 6889, 6892, 6893, 6894, 6895, 6896, 6897] still not executed: NONE Controls re-run with case w present: revert the actuator guard -> v fails revert started_within's UNKNOWN returns -> w fails, naming all three (undecodable, missing, non-numeric) restored -> 24/24 Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * fix(health-check): an UNKNOWN probe inherited the destructive confirm window qingyun-wu's follow-up blocker, reproduced at the exact head before fixing. My first fix suppressed the restart on UNKNOWN but DELIBERATELY left the confirmation window intact, reasoning that an intermittently-failing probe would otherwise reset it forever and silently disable recovery. I flagged that to the reviewer as a judgment call. It was the wrong call, and the repro is theirs: liveness dead -> UNKNOWN -> dead at t=1000000/1000200/1000400 before observed / probe-failed / RESTARTED restart_calls == [False] after observed / probe-failed / observed restart_calls == [] A single post-uncertainty DEAD reading inherited a window opened BEFORE the uncertainty, so "the death persists across a pass" did not hold — the middle pass confirmed nothing at all. Both directions fail, so the only question is which way. A probe too flaky to give two consecutive affirmative deaths means the state is genuinely unknown, and NOT restarting is correct there. Killing a healthy core on one observation plus a gap is not. **This is the same wrong-direction call I made earlier in this PR** — weighing "recovery disabled" over "healthy core killed" when the actuator's action is destructive. UNKNOWN now invalidates the streak: the confirmation window resets, and recovery resumes as soon as two clean consecutive observations exist. Case `x` asserts BOTH halves, because the first alone would also pass if I had simply disabled recovery: 1. dead -> UNKNOWN -> dead must NOT restart 2. then dead -> dead (probe clean) MUST restart Controls run, each discriminating: revert ONLY the reset -> x fails, naming the inherited window disable recovery outright -> x fails on its PAIRED half (plus j, s, u), which is exactly what that half exists to catch restored -> 25/25 Also ran all 51 `tests/health-check*.test.py` suites: 0 failing. Still outstanding and owner-gated: the supervised live restart round trip. Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> * test(recover-core): adopt the no-argument restart contract from #2742 This branch's dead-relaunch cases were written against restart_fn(standard_context); #2742 removed the model downgrade and the parameter with it. Case e already asserted the no-argument form, so the file held both generations at once. Stand: Echo Act IV Pro * fix(health-check): declare the three-state contract, cut comments to invariants Both local helpers return None for UNKNOWN and the actuator branches on it, so -> bool understated the contract that makes the fail-closed path work. The comment blocks restated the code around them. Each keeps the invariant a reader cannot see: the three states, why a mode flip resets the window, why an absent mode implies a wedge. Stand: Echo Act IV Pro * fix(health-check): keep the four cited comments inside the two-line cap Drops the narration from three test docstrings and the wedge/dead gating comment, keeping each stated constraint. No behaviour change; the reset triggers named in the src comment are unchanged. Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * docs(test): bring core-recovery docstrings to the two-line comment cap Twelve over-length docstrings, not the four cited — the review said "including", so I scanned rather than fixing only the named lines. The largest were a 35-line module header and a 23-line case narration. What went: the incident replay, the a)-n) case table (which restated the test function names), and the reasoning-history prose. What stayed: the constraint each case exists to pin, in at most two lines. The removed reasoning is real and is moved to the PR body rather than dropped — particularly case_x's "both halves are asserted, because the first alone would also pass if recovery were simply disabled", which is the non-obvious part. Scope: added lines only. The one remaining 3-line `#` run (L168) is pre-existing and untouched. Verified: all core-recovery invariants pass; a validated AST scan (control: a known 3-line docstring is detected) now reports 0 over-length docstrings. Stand: Echo Act IV Pro Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com> * test(health-check): assert --recover-core stays out of the shipped fallback plist The template's own #2246 note records that `--recover-core` was REMOVED from the default args, and until now nothing defended it: re-arming the flag would land green and make a deliberately-stopped core relaunchable. Asserts against the REAL committed plist, not the inline TEMPLATE — a fixture assertion would certify nothing about what ships. Why the comment strip: the template's doc-comment contains `--emit-task`, `--recover-core` etc, and XML forbids `--` inside a comment, so expat (plistlib) rejects the file at line 24 while macOS `plutil -lint` accepts it. That asymmetry is why the shipped file had never been asserted against. Controls both ways: passes on the committed file; planting the flag into a copy of that same template yields args ending `--recover-core` and the assertion fires. Stand: Echo Act IV Pro * style(tests): cut the two new comment blocks to the 2-line cap CI's `prose-cap` flagged them at 7 and 5 lines. The narrative belongs in the PR body, which already carries it. Stand: Echo Act IV Pro * fix(tests): case u pinned .alive mtime to the real clock, not the synthetic one main made _core_started_within's freshness check two-sided (heartbeat_is_fresh, [-5.0, 90.0)) where the branch only skipped stale files. The fixture wrote peer-host.alive at wall-clock mtime while asserting against now=1_000_100, so the file read as future-dated and was skipped -- the peer vanished and the precondition failed. Passes on the branch, fails on the merge. Stand: Echo Act IV Pro * fix(tests): make the --recover-core positive control able to fail The old control asserted "--recover-core" in (_args + ["--recover-core"]), which is true by construction and never touches the loader. Plant the flag in a real template, render it through the same parser, and assert both that the loader sees it and that the absence check goes red. Addresses review P2. Stand: Echo Act IV Pro * style(health-check): cut the recover-core trigger comment to the 2-line cap main's review-checks carries prose-cap; the branch's copy did not, so the 4-line block only became visible once the branch was brought up to date. Both trigger conditions and the two-observation requirement are kept. Stand: Echo Act IV Pro * fix(health-check): the local-core probes accepted a future-dated heartbeat qingyun-wu's P1, independently confirmed by bassilkhilo-ag2 and reproduced here before fixing. `_local_core_alive` and `_local_core_started_within` used one-sided age tests: return (time.time() - alive_file.stat().st_mtime) < max_age_s if now - alive_file.stat().st_mtime >= 90.0: return False A future-dated mtime (clock step, bad write, skewed sync) has a NEGATIVE age, which both accept. The core then reads as permanently alive AND permanently just-booted, suppressing the dead-core recovery this PR exists to add. `heartbeat_is_fresh()` in the same module is two-sided for exactly this reason and its own docstring says so; these two call sites did not use it while their fleet-wide siblings did. Reproduced with a heartbeat dated +100000s: before _local_core_alive=True _local_core_started_within=True after _local_core_alive=False _local_core_started_within=False heartbeat_is_fresh said False throughout New test, validated BOTH ways: 5/5 pass at this head, 2 FAIL against the pre-fix code (the two future-mtime cases). Separate file from heartbeat-freshness-bounds.test.py on purpose: that suite covers `_live_core_socket` / `_local_core_socket`, not these two, and returns rc=0 against the broken code -- so its name ("bounded on both ends") reads like coverage these functions never had. Measured, not assumed. Negative controls included so the fix cannot over-correct: fresh stays alive/just-booted, stale stays dead, missing file stays dead not UNKNOWN. Stand: Echo Act IV Pro * fix(recover-core): graceful-stop tombstone gates the dead-core relaunch A SIGTERM/SIGINT'd heartbeat now writes a {label}.stopped tombstone before unlinking .alive; the next heartbeat run clears it. recover_core_if_wedged checks the tombstone (injectable stopped_fn) before the destructive dead branch and returns action=deliberate-stop without restarting or paging — a relaunch must not undo an intentional stop (john-the-dev, 17:26Z). Also homes the flag-enable gate text beside the #2246 NOTE in the fallback plist — the one place whoever re-adds --recover-core will see it. Test: case k (tombstoned dead core -> deliberate-stop, zero restarts, zero pages) with a no-tombstone control that still relaunches. Stand: Echo Act IV Pro * test(recover-core): collect the tombstone case; cover signal handler and helper The deliberate-stop test was defined but never in the runner's explicit cases list — a pre-existing case_k name collision meant 'case k ✓' printed while the new test never ran, and CI's diff coverage caught line 10612 unexecuted. Renamed to case_k2 and wired it, added case_k3 driving _local_core_stopped directly (tombstone present/absent/unresolvable label), and a heartbeat test proving _handle_signal writes the tombstone before unlinking .alive. Stand: Echo Act IV Pro * fix(stop-core): canonical stop path publishes the graceful-stop tombstone qingyun-wu's P1 on 5b1e461: stop-core.sh kills tmux sessions and never signals the heartbeat sidecar, so the sidecar's core-gone exit unlinked .alive without a tombstone and 'stop core' stayed indistinguishable from a crash. mark_stopped() is now the one shared writer (mkdir included — the e2e run caught a fresh-workspace stop failing on a missing state/cores/, previously swallowed by the handler's best-effort except), called by both the signal handler and a new --mark-stopped CLI mode that stop-core.sh invokes before killing sessions — and only on the path that actually stops one, so probing a crashed core cannot mask it from recovery. End-to-end contract in tests/stop-core.test.sh: real stop-core.sh run publishes the tombstone; recover_core_if_wedged with its DEFAULT production stopped_fn returns deliberate-stop; crash control without the tombstone still relaunches; nothing-to-stop path writes no tombstone. Stand: Echo Act IV Pro * test(stop-core): scope every run to a throwaway workspace d3a83e1 made every stop-core.sh invocation call --mark-stopped, so the pre-existing unscoped cases wrote a real tombstone into the checkout's workspace — health-check's default stopped_fn then read it and case x reported deliberate-stop where a restart was required (the CI failure at that head). Export SUTANDO_TEST_MODE + throwaway SUTANDO_WORKSPACE + pinned host label for the whole file; the e2e case layers its own. Stand: Echo Act IV Pro * style(core-heartbeat): trim mkdir comment to the 2-line cap Review-checks prose-cap flagged the 3-line block at mark_stopped; the narrative lives in the PR body, the constraint stays in the code. Stand: Echo Act IV Pro --------- Co-authored-by: john-the-dev <52230987+john-the-dev@users.noreply.github.com> Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Chi Wang <sonichi@users.noreply.github.com>
…CLI, visibly (#3898) * feat(scripts): switch-model.sh — change the core's model without the CLI, visibly Owner, 2026-09-04: "can we make it easy to change model without using the CLI?" Today the only switch is `/model` typed into the core's terminal; it persists by writing `model` into settings.json, which the launcher inherits (it passes no --model). `scripts/switch-model.sh <model>` does both halves from outside the CLI: pins `model` in `$CLAUDE_CONFIG_DIR/settings.json` (atomic, other keys kept) for the next launch, and sends `/model <name>` + Enter to the live `sutando-core` tmux pane (same path the menu-bar app uses for `watcher`). Every switch writes `<workspace>/state/model-switch.json` with the previous model and a UTC timestamp — the #2742 lesson: a pin nothing can see becomes a permanent downgrade. Names are an allowlist (the CLI's aliases or a claude-* id with optional [1m]); anything else is refused before any write. Exit 3 reports a pin with no live pane, so a caller cannot mistake "next launch" for "now". tests/switch-model.test.sh: 11 checks with a tmux PATH shim — refusal is side-effect free; settings pinned with other keys intact; record carries model/previous/ts/writer; the pane receives the literal command then Enter on the core's socket; missing session → rc 3 with the pin landed; --dry-run touches nothing. Mutations: dropping the Enter keystroke reddens check 6 only; dropping the record write reddens check 5 only. No shellcheck on this host; CI collects *.test.sh. Stand: Echo Act IV Mini * fix(switch-model): resolve settings.json through the claude-home-path helper, not an inline fallback The repo's guard refuses a new `${CLAUDE_CONFIG_DIR:-$HOME/.claude}`; the helper carries the same resolution order and the deprecation banner. The 11-check suite is unchanged and green (it sets CLAUDE_CONFIG_DIR, which the helper honors first). Stand: Echo Act IV Mini * refactor(model-switch): package as a skill; record before pin; canonical socket + python resolvers Addresses qingyun-wu's review on #3898, all four points: 1. Targeting: settings via `sutando-config.sh claude-home-path`, socket via `sutando-config.sh tmux-socket` when SUTANDO_TMUX_SOCKET is unset, session `sutando-core` unless SUTANDO_TMUX_SESSION / --session; explicit flags still win. 2. Partial failure: the switch record is written FIRST; a failed pin removes the record and reports "nothing changed" (exit 1). The reviewer's case (state path is a file) now leaves settings.json untouched — test 12/13. 3. Python from `sutando-config.sh python-bin`; no bare python3 (test 14). 4. Capability lives in skills/model-switch/ (SKILL.md + scripts/); scripts/switch-model.sh is a thin exec. Live witness, real tmux pane on a throwaway socket, real script, no shim: pane received "/model fable" + Enter; settings pinned opus -> fable; record written first. (The core's own pane is not used: that would switch this session's model mid-task.) Suite: 15 checks (11 prior + 4 new). tmux send-keys targets the session by name as the menu-bar app does; `=name` is not a valid pane target. Stand: Echo Act IV Mini * fix(model-switch): read the input box before typing, and refuse on a Codex runtime Owner questions (2026-09-04): does it work for Claude Code only, and does it check whether the input box is clear? - Claude Code only, now enforced: when `sutando-config.sh core-runtime` is codex the script refuses (exit 4) and names the real path — the Codex core takes its model as a launch argument (`codex -m` from SUTANDO_CORE_MODEL), so a change there is an owner-gated restart. - The input box is read before anything is written: `capture-pane`, take the prompt line (`❯`/`> `), and if it carries text refuse (exit 5, quoting it) — the same buffer check the menu-bar app runs before it sends `watcher`. A clear prompt with a turn running still sends; the CLI queues the command and switches when the turn ends, and the success line says so. - `--dry-run` reports before the preflight and touches nothing. Tests 16–19 pin all three; 19 checks. Live witness re-run on a real pane: received "/model fable". Stand: Echo Act IV Mini * fix(model-switch): defaults from the runtime descriptor; a failed pin keeps the prior record qingyun-wu's second round on #3898: 1. Targeting — brain, socket and session now come from `sutando-config.sh runtime` (runtime-authored, foreign-caller safe: the socket is read from the core's heartbeat, the brain is the configured Sutando config dir), never from ambient env or ~/.claude. `--brain/--socket/--session` remain explicit operator overrides; the test suite uses them. 2. The switch record is STAGED beside the prior one and committed only after the pin succeeds; a failed pin discards the staged file and the last trusted record is byte-intact. Regression 20–22 starts with an existing record and forces the replace to fail (settings.json made a directory, the reviewer's control). 22 checks. Witness on a real throwaway pane with a prior record present: pane received "/model fable", record replaced only after the pin. The restart round trip on a safe host is still owed and is an owner ask. Stand: Echo Act IV Mini * test(model-switch): the descriptor read goes through tmux too — key 'nothing sent' on send-keys; assert the descriptor call Follow-up to 5a82f79, which was committed with these two checks red: the runtime-descriptor read calls tmux (has-session on the socket) so an empty shim log was the wrong 'nothing sent' predicate, and check 15 still looked for the retired tmux-socket getter. 22/22 green. Stand: Echo Act IV Mini * fix(model-switch): the record commit is inside the transaction; descriptor fields survive spaces qingyun-wu round 3 on #3898: 1. The final record commit sat outside the guarded block, so a record path that is a directory left settings.json pinned and unrecorded — the one state the transaction forbids. Now: the prior settings bytes are kept, and if the staged record cannot be committed the settings are rolled back to those exact bytes (tests 23/24: record path is a directory → settings byte-identical to before, nothing sent, message says so). 2. The descriptor was parsed with one whitespace-delimited `read`, which split a brain under "Application Support". Fields now travel one per line and are read whole-line; `--descriptor-file` is an explicit override so the suite can inject a descriptor (test 25: brain, socket and session with spaces arrive intact, with the ambient env unset, since an explicit env override still wins by design). 25 checks; committed only after the suite's exit code was read. Witness on a real throwaway pane: received "/model fable". Stand: Echo Act IV Mini * feat(scripts): tmux-send-line.sh — one sender for lines typed into a core pane; the app uses it Owner, 2026-09-04: "Is there duplicate code in checking input box and typing?" Yes: the menu-bar app carried has-session + capture-pane + last-❯-line + send-keys in Swift (`tmuxSendKeys`, `watcherKeystrokesQueued`), and the model-switch script re-implemented the same policy in bash. Two copies of one policy is the defect the architecture rules name. `scripts/tmux-send-line.sh <session> <line> [--socket S] [--refuse-if-pending] [--skip-if-queued WORD] [--dry-run]` now owns it: resolves tmux (Homebrew, /usr/local, PATH), checks the session, reads the CURRENT prompt (the last ❯ line; one space or nbsp after the glyph is not input), applies the policy, sends the line literally (-l) then Enter. Exit 0 sent · 3 no session · 4 no tmux · 5 pending text · 6 word queued. The app's two functions are replaced by `tmuxSendLine(session:line: skipIfQueued:)`, which execs the script from `repoRoot` with the app's socket; the watcher path keeps its three outcomes (queued → skip, sent → notify, else → fallback notice). Built here with the README's swiftc line (rc=0); `restart-coordinator-swift` and `sutando-config-swift` suites OK. tests/tmux-send-line.test.sh against a REAL tmux server on a throwaway socket: R1 literal line + Enter delivered; R2 missing session → 3; R3 a pane whose prompt carries text → 5 quoting it; R4 --skip-if-queued on the queued word → 6; P1–P3 the prompt parser (last ❯ wins; nbsp-empty reads empty; leading whitespace ignored). The model-switch script (#3898) switches to this sender once it lands. Stand: Echo Act IV Mini * fix(model-switch): one lock across preflight, transaction and send; SKILL.md matches the code qingyun-wu round 4 on #3898: 1. Overlapping invocations could interleave the settings write, the record commit and the live send (measured: settings haiku, record sonnet). A flock on `<state>/.model-switch.lock` (fd 9, held for the process lifetime) now spans preflight, the whole transaction and the send. Tests 26/27: two concurrent switches with a delayed capture — settings, record and the last live command agree; the sends do not interleave. A lock path that cannot be opened (state path is a file) refuses with "nothing changed" — the earlier partial-failure check caught the bash redirection error that replaced that message. 3. SKILL.md rewritten to the actual ordering: preflight → prior bytes read → record staged → pin → record committed (settings rolled back on commit failure) → send; defaults from the runtime descriptor. 2. The safe-host restart witness remains the owner's ask, stated as such. 27 checks; committed after the suite's exit code was read. Stand: Echo Act IV Mini * fix(tmux-send-line): fail closed on a failed inspection; one lock per socket+session; shim suite always runs qingyun-wu on #3899: 1. A failed capture-pane or a missing/failing parser interpreter read as an empty prompt and the send went ahead. Now: `set -o pipefail`, the interpreter is checked before use, capture and parse are checked separately, and any failure refuses with exit 7. Controls S5 (capture fails) and S6 (interpreter fails): nothing sent. 2. Two callers could interleave payloads before either Enter. A flock on a per-socket+session lock file (fd 9) now spans inspection and both send-keys. Control S7: two overlapping callers with a delayed capture serialize to lit/enter/lit/enter. 3. The suite's assertions were gated on tmux living at two fixed paths. The shim leg (S1–S7) now runs unconditionally through the PATH shim the resolver finds first; the real-tmux leg (R1–R4) is the optional second leg and runs here. Swift unchanged from 220e684 (built there, rc=0). Stand: Echo Act IV Mini * refactor(model-switch): the live half goes through the shared sender; stacked on #3899 qingyun-wu round 5 on #3898: two locks on one pane are not one lock. The switch script no longer touches tmux itself — the preflight is the shared sender's --dry-run (--refuse-if-pending, under its socket+session flock) and the send is the shared sender proper, so a model command and the app's watcher line cannot interleave. Local has-session/capture-pane/ send-keys: 0 remaining. The switch keeps its own transaction lock for settings+record. Test 28 (cross-sender): a switch and a watcher line through the same sender with a delayed capture serialize to lit/enter/lit/enter. 28 checks, committed after the suite's exit code was read. Merge order: #3899 first (this branch's base), then this. Stand: Echo Act IV Mini * model-switch: drop the settings.json pin — the CLI persists /model itself Owner, 2026-09-04: "isn't the PR about changing model without restart?" It is. The pin was the one thing in the script that touched a launch-time path, and it duplicated a write Claude Code already makes: measured on this host, settings.json's `model` key changed at 14:08Z, hours into a session, with no writer in the repo — the CLI's own /model wrote it. So the script now records the switch (previous model read from settings.json, read-only) and types /model through the shared sender under its lock; it never writes settings.json, and there is no launch-time claim left to witness across a restart. Suite rewritten to the narrowed contract: 19 checks, including "settings.json is byte-identical after every case" (3, 19) and "the script touches no tmux itself" (15); the race (16/17) now asserts record == last live send; cross-sender (18) kept. SKILL.md and the catalog entry say what the script does now. Committed after the suite's exit code was read. Stand: Echo Act IV Mini * fix(model-switch): the runtime gate fails closed — an unresolved or unknown runtime refuses before any record qingyun-wu round 6 on #3898: `core-runtime` failing or printing nothing was read as claude, and SUTANDO_CORE_RUNTIME=bogus still wrote a record (rc=3). Now a nonzero resolver, an empty value, or a runtime other than claude/codex refuses with exit 4 before the record or any pane action. Tests 20/21 pin both shapes (a bogus value, which the resolver itself rejects; a resolver that exits 1 via a wrapper repo). 21 checks. Stand: Echo Act IV Mini * fix(model-switch): the send is initiation, not success — record only after the CLI accepts qingyun-wu round 7 on #3898 (P1): on a warm core the CLI shows a confirm dialog, the old script exited 0 and wrote the record while the model had not changed. Now: baseline the acceptance lines on screen, send, then wait for a NEW 'Set model to' line (pane-observe.sh, the skill's only pane reader). The dialog is answered only under --confirm (an owner instruction is the authorization); otherwise it is cancelled with Escape and the script exits 6 with nothing recorded. No acceptance within --accept-timeout (20s) is exit 8, nothing recorded. previous falls back to the last record when settings.json carries no model key and names its source. Controls 22-26; 26 checks. Parent control: same dialog shim, the old script exits 0 and writes the record. Stand: Echo Act IV Mini * fix(model-switch): accept only the requested model, and snapshot previous before the send qingyun-wu round 8 on #3898, two P1s. (1) previous was read after the CLI accepted — and persisted — the new model, so a normal switch to opus recorded previous=opus; it is now snapshotted before the send and carried to the record. (2) the observer counted any 'Set model to' line; it now matches the REQUESTED model's display name (family + version derived from the alias or claude-<family>-<ver> id), so another model's acceptance in the polling window is exit 8, nothing recorded. Headers cut to two lines. Fixture: the shim renders the real CLI display name per accepted send, TMUX_ACCEPT_AS forces a wrong model, TMUX_PERSIST_SETTINGS mimics the CLI saving the pick. Controls 27-29; 29 checks. Parent controls: wrong-model rc=0 + record; persist -> previous=opus. Head: rc=8 + none; previous=fable. Stand: Echo Act IV Mini * fix(model-switch): match the exact display name, and refuse when the pane cannot be read qingyun-wu round 9 on #3898, two P1s. (1) 'Set model to Opus 5' as a bare substring also matched 'Opus 5.1', so a request for one version could accept its neighbour; the acceptance regex now ends at a boundary (a versioned id matches only its version; an alias accepts the family at any version). (2) a failed capture-pane read as count 0 with exit 0, so a later stale line could pass as new; the observer now propagates the failure and switch-model refuses BEFORE sending. Controls 30-33; 33 checks. Parent: adjacent version rc=0 + record; failed baseline proceeds to send. Head: rc=8 + none; rc=7, nothing sent. Stand: Echo Act IV Mini
Why
Owner directive, 2026-08-08: "we need to removed it!"
A second wedge-restart inside one hour pinned
SUTANDO_CORE_MODEL=opus, moving the core to the standard 200K window so it kept working instead of re-wedging. The escalation worked. Its lifetime did not:setenv -uin the repo is inside an f-string — the text of a warning, not a command anything runsstart-cli.shre-supplies the var into the new session andnew-session -Adiscards its own-eSo one recovery became a permanent downgrade. Measured on a peer core in #2717: 17 days on the pin, 165 autocompactions in 25h — one every 9.1 minutes — found only because the owner noticed the symptom and asked. That has now happened twice, which is what prompted this.
Why the two previous attempts didn't hold
--recover-coreauto-restart is unreliable: single soft signal triggers a destructive restart (#1428) #2246 removed--recover-corefrom the launchd template default. That stops the automatic trigger on new installs only — editing a template does not rewrite a plist already on disk.This removes the mechanism instead.
What changes
_default_core_restart()takes no argument and sets no model.recover_core_if_wedgeddropsstandard_context/mode/ctx_note. The owner DM read "Auto-restarting in standard 200K context (the 1M restart didn't hold)" — with the pin gone that sentence would be false, so it goes too, along with themodekey andlast_restart_mode.RECOVER_MAX_PER_HOUR+ the existing give-up DM remain the bound on a re-wedging core. That backstop is what made the model escalation optional in the first place: a pathological wedge still stops after N restarts and pages the owner. Removing the downgrade does not open an unbounded restart loop.core-model-pinprobe STAYS, re-worded. Nothing sets the pin now, so a pin means legacy-or-manual — exactly the case still suspected on a peer host, and the probe is how anyone finds it.Tests — inverted, not deleted
Both suites asserted the old behaviour, so removing it had to flip them:
case_galso now asserts the DM no longer promises a downgrade ("200K"/"didn't hold"must not appear), because a stale reassuring message is its own defect.At the parent commit the new tests give 6 failures + 1 error:
At this head:
health-check suites: pass=60 fail=0.Live-path evidence — real core restart on this exact head (2026-08-08T17:20Z)
Closes the live-path gate. Exercised on the running core at
c7dad2d76230aa7653ce977fea03938df4289049, which is this PR's exact head — not a lookalike build.Before the restart, the pin was latent but armed.
SUTANDO_CORE_MODEL=opussat in tmux's global scope while the running core's argv carried no--model, so the pin could only bite on the nextnew-session— which is exactly what a restart performs. That makes it a clean discriminator: on the pre-fix codestart-cli.shre-supplies the var from its launching env, so the new core must come up pinned.After the restart:
Probe verdict flipped at the owner-visible endpoint:
Durability check. The checkout was then restored to
main(d709f4a7) and the probe re-run there: stillok, both tmux scopes still clear. So the cleared state is a real state change, not an artifact of standing on the PR head.Full before/after capture:
workspace/notes/2742-restart-proof-2026-08-08.md.Still not verified by me
No wedge was induced. The restart above was a real
start-cli.sh --restartthrough the normal launcher path, which is what exercises the-e-discardingnew-session -Abehaviour this PR targets;recover_core_if_wedged's own decision logic is still exercised through the injectedrestart_fn. A queued-task/result round trip did run across the restart — the post-restart core drained four owner tasks fromtasks/and wrote theirresults/files.@sonichi