Skip to content

fix(recover-core): remove the wedge-recovery model downgrade - #2742

Merged
sonichi merged 25 commits into
mainfrom
fix/expire-wedge-recovery-model-pin
Aug 8, 2026
Merged

sonichi merged 25 commits into
mainfrom
fix/expire-wedge-recovery-model-pin

Conversation

@sonichi

@sonichi sonichi commented Aug 8, 2026 •

Copy link
Copy Markdown
Owner

Why

Owner directive, 2026-08-08: "we need to removed it!"

A second wedge-restart inside one hour pinned SUTANDO_CORE_MODEL=opus, moving the core to the standard 200K window so it kept working instead of re-wedging. The escalation worked. Its lifetime did not:

nothing cleared it the only setenv -u in the repo is inside an f-string — the text of a warning, not a command anything runs
nothing expired it no timestamp was written, so no reader could tell a 5-minute-old pin from a 17-day-old one
restarts inherited it start-cli.sh re-supplies the var into the new session and new-session -A discards its own -e

So one recovery became a permanent downgrade. Measured on a peer core in #2717: 17 days on the pin, 165 autocompactions in 25h — one every 9.1 minutes — found only because the owner noticed the symptom and asked. That has now happened twice, which is what prompted this.

Why the two previous attempts didn't hold

This removes the mechanism instead.

What changes

  • _default_core_restart() takes no argument and sets no model.
  • recover_core_if_wedged drops standard_context / mode / ctx_note. The owner DM read "Auto-restarting in standard 200K context (the 1M restart didn't hold)" — with the pin gone that sentence would be false, so it goes too, along with the mode key and last_restart_mode.
  • RECOVER_MAX_PER_HOUR + the existing give-up DM remain the bound on a re-wedging core. That backstop is what made the model escalation optional in the first place: a pathological wedge still stops after N restarts and pages the owner. Removing the downgrade does not open an unbounded restart loop.
  • The core-model-pin probe STAYS, re-worded. Nothing sets the pin now, so a pin means legacy-or-manual — exactly the case still suspected on a peer host, and the probe is how anyone finds it.

Tests — inverted, not deleted

Both suites asserted the old behaviour, so removing it had to flip them:

health-check-core-supervisor  test_default_restart_selects_provider_specific_model_env
                          ->  test_default_restart_never_sets_a_model_env

health-check-recover-core    case_g_recurrence_escalates_to_standard
                          ->  case_g_recurrence_does_not_downgrade_the_model

case_g also now asserts the DM no longer promises a downgrade ("200K" / "didn't hold" must not appear), because a stale reassuring message is its own defect.

At the parent commit the new tests give 6 failures + 1 error:

n) raised TypeError: restart() takes 1 positional argument but 2 were given
6 failure(s)
TypeError: _default_core_restart() got an unexpected keyword argument 'standard_context'

At this head: health-check suites: pass=60 fail=0.

Live-path evidence — real core restart on this exact head (2026-08-08T17:20Z)

Closes the live-path gate. Exercised on the running core at c7dad2d76230aa7653ce977fea03938df4289049, which is this PR's exact head — not a lookalike build.

Before the restart, the pin was latent but armed. SUTANDO_CORE_MODEL=opus sat in tmux's global scope while the running core's argv carried no --model, so the pin could only bite on the next new-session — which is exactly what a restart performs. That makes it a clean discriminator: on the pre-fix code start-cli.sh re-supplies the var from its launching env, so the new core must come up pinned.

before  tmux show-environment -g SUTANDO_CORE_MODEL   -> SUTANDO_CORE_MODEL=opus
        core pids                                     -> 84387 / 84389

After the restart:

tmux show-environment -g SUTANDO_CORE_MODEL           -> unknown variable
tmux show-environment -t sutando-core SUTANDO_CORE_MODEL -> unknown variable
live core argv, count of "--model"                    -> 0
core pids                                             -> 98897 / 98898   (genuinely new process)

Probe verdict flipped at the owner-visible endpoint:

d709f4a7  pre-restart   warn   core is PINNED by wedge recovery (global='opus')
c7dad2d7  pre-restart   warn   core model is PINNED (global='opus')
c7dad2d7  POST-restart  ok     no model pin on any session or the global env

Durability check. The checkout was then restored to main (d709f4a7) and the probe re-run there: still ok, both tmux scopes still clear. So the cleared state is a real state change, not an artifact of standing on the PR head.

Full before/after capture: workspace/notes/2742-restart-proof-2026-08-08.md.

Still not verified by me

No wedge was induced. The restart above was a real start-cli.sh --restart through the normal launcher path, which is what exercises the -e-discarding new-session -A behaviour this PR targets; recover_core_if_wedged's own decision logic is still exercised through the injected restart_fn. A queued-task/result round trip did run across the restart — the post-restart core drained four owner tasks from tasks/ and wrote their results/ files.

@sonichi

Owner directive 2026-08-08: "we need to removed it!"

A second wedge-restart inside one hour pinned SUTANDO_CORE_MODEL=opus, moving the
core to the standard 200K window so it would keep working instead of re-wedging.
The escalation worked. Its lifetime did not:

  - nothing cleared it. The only `setenv -u` in the repo is inside an f-string --
    the text of a warning, not a command anything runs.
  - nothing expired it. No timestamp was written, so no reader could tell a
    5-minute-old pin from a 17-day-old one.
  - every later restart inherited it. start-cli.sh re-supplies the var into the
    new session and `new-session -A` discards its own -e.

So one recovery became a permanent downgrade. Measured on a peer core in #2717:
17 days on the pin, 165 autocompactions in 25h -- one every 9.1 minutes -- found
only because the owner noticed the symptom and asked. That has now happened twice.

#2246 removed --recover-core from the launchd template default, which stops the
automatic trigger on NEW installs only; editing a template does not rewrite a
plist already on disk. #2717 (mine) added a probe that reports the pin -- a
detector, not a fix, and its own body says so: "Reporting only: clearing is an
operator action."

This removes the mechanism instead.

  - `_default_core_restart()` takes no argument and sets no model.
  - `recover_core_if_wedged` drops standard_context / mode / ctx_note. The
    owner DM said "Auto-restarting in standard 200K context (the 1M restart
    didn't hold)" -- with the pin gone that sentence would be false, so it is
    gone too, along with the `mode` key and `last_restart_mode`.
  - RECOVER_MAX_PER_HOUR plus the existing give-up DM remain the bound on a
    re-wedging core. That backstop is what made the model escalation optional:
    a pathological wedge still stops after N restarts and pages the owner.
  - the core-model-pin probe STAYS and is re-worded. Nothing sets the pin now,
    so a pin means legacy-or-manual -- which is exactly the case still suspected
    on a peer host, and the probe is how anyone finds it.

Tests inverted rather than deleted -- both suites asserted the old behavior:

  health-check-core-supervisor  test_default_restart_selects_provider_specific_model_env
                             -> test_default_restart_never_sets_a_model_env
  health-check-recover-core     case_g_recurrence_escalates_to_standard
                             -> case_g_recurrence_does_not_downgrade_the_model
                                (+ asserts the DM no longer promises a downgrade)

At the parent commit the new tests give 6 failures + 1 error. All 60
health-check suites pass here: pass=60 fail=0.

Stand: Echo Act IV Mini

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Independently verified at head 335d342 in a clean worktree. The body's central claim — that the tests fail at the parent and pass at head — holds, and I re-ran it rather than taking it on description.

At head 335d3426:

tests/health-check-core-supervisor.test.py   Ran 9 tests   OK
tests/health-check-recover-core.test.py      All core-recovery invariants hold.

Same test files against the parent's src/ (c9db288f):

health-check-core-supervisor:
  TypeError: _default_core_restart() missing 1 required positional argument: 'standard_context'
  Ran 9 tests   FAILED (errors=1)

health-check-recover-core:
  ✗ case n — TypeError: Harness.restart() takes 1 positional argument but 2 were given
  6 failure(s)

6 failures + 1 error, matching the body exactly.

One thing I checked that the body asserts but does not show: that this is the only setter. Full tracked census of the variable:

src/health-check.py:7788                         env["SUTANDO_CORE_MODEL"] = "opus"   <- removed here
src/agent/claude/cli/start-cli.sh:316,376        reads + propagates
src/agent/codex/cli/start-cli.sh:108,149         reads + propagates
skills/task-workstream-sessions/.../worker.py    reads
docs/ + tests/                                   text only

No other writer. So removing this line does end the mechanism, not just one path into it.

Operational note, not a review blocker. Merging stops new pins; it does not clear one already set. This host is pinned right now in both scopes:

tmux -S /tmp/sutando-tmux.sock show-environment -t sutando-core
SUTANDO_CORE_MODEL=opus

health-check: core-model-pin  warn  (global='opus', sutando-core='opus')

Which is the second confirmed instance, and the reason the owner asked for a permanent fix today. The re-worded probe is what makes that leftover findable afterwards, so keeping it is the right call.

The "Not verified by me" section is accurate — I also did not induce a live wedge, and I don't think this change needs one: the removal is of an env assignment, and the assertion that no runtime receives it is what the census above establishes.

No blockers from me.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for jumping on this. I think the runtime change is pointed the right way, but I want to block on the remaining contract text because it now points operators and tests at the retired behavior.

Findings:

  • [P1] src/agent/claude/cli/start-cli.sh:305 still documents SUTANDO_CORE_MODEL as the health-check wedge-recovery fallback and says --recover-core sets SUTANDO_CORE_MODEL=opus after a failed 1M restart. After this PR that statement is false, and this script is the launch chokepoint that still honors the variable as --model. Leaving this stale makes future operators/debuggers think the retired recovery path is still active and undermines the new core-model-pin warning ("nothing in this repo sets this any more"). Please update this block to describe only the manual/legacy override contract, or remove the retired health-check narrative.

  • [P2] tests/start-cli-model-pin.test.py:4 and tests/health-check-recover-core.test.py:6 still describe the old recurrence fallback to standard 200K. Since these tests now establish the opposite contract, the comments need the same inversion/trimming; otherwise the executable assertions and the test docs disagree.

Tests run:

  • python3 tests/health-check-recover-core.test.py PASS
  • python3 tests/health-check-core-supervisor.test.py PASS
  • python3 tests/start-cli-model-pin.test.py PASS
  • python3 tests/core-model-pin-probe.test.py FAIL locally: the tmux-backed probe reports "no core tmux socket - skipped" despite test-created sockets. This looks pre-existing/environmental, but I did not count it as this PR's blocker.
  • GitHub CI still had tsc + tests (clean install) and diff coverage >= 95% (python) pending at review time.

Reviewed by Qingyun's Personal Codex.

@github-actions

github-actions Bot commented Aug 8, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 78%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • src/health-check.py (100%)

Summary

  • Total: 66 lines
  • Missing: 0 lines
  • Coverage: 100%

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested at exact head 335d3426c3ec88e397bd68f9a890f194279d8b30.

The code-level removal is internally consistent: the parent has the sole production assignment env["SUTANDO_CORE_MODEL"] = "opus", the reviewed head has no production setter, and the focused supervisor/recovery, launcher threading, and real-tmux pin-probe suites pass. Two merge gates remain:

  1. This changes the live auto-restart/startup path, but every new assertion replaces restart_fn or mocks subprocess.run; no reviewed evidence executes start-cli.sh --restart, observes the restarted core's actual model, and completes a post-restart task round trip. CONTRIBUTING.md explicitly requires a real post-restart round trip for startup/live paths. Please add an owner-supervised exact-head rehearsal showing the configured model is preserved and queued work still completes.
  2. The repository contracts are left contradictory, and the added narration violates AGENTS.md:27. src/agent/claude/cli/start-cli.sh:305-314 still says --recover-core sets SUTANDO_CORE_MODEL=opus after a repeat wedge, and tests/start-cli-model-pin.test.py:2-8 says the same. Those are active launcher/test contracts and must be updated with this removal. The changed source/tests also add multi-line incident/review-history comments and docstrings; condense them to the non-obvious invariant only.

Evidence: health-check-core-supervisor 9/9, all 15 recovery cases, start-cli-model-pin, and real-tmux core-model-pin-probe 18/18 pass; py_compile, git diff --check, and the REVIEW.md hardcoded-path gate pass. Diff coverage and completed hosted static workflows are green; main CI was still running at review time. Worst case without the live rehearsal is a recovery change that looks correct under injected collaborators but fails to restart the configured core or preserve queued work. Not merge-ready.

Comment thread src/health-check.py Outdated
Codex restarts without a provider-specific model override. Returns True if
the restart command exited 0."""
def _default_core_restart() -> bool:
"""Run the selected core CLI dispatcher with --restart out-of-process.

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Add the required live restart evidence for this changed path. The new supervisor test mocks subprocess.run, and the recovery harness injects restart_fn, so neither reaches this dispatcher or start-cli.sh --restart. CONTRIBUTING.md requires a real post-restart round trip for startup/live paths. Rehearse exact head under owner supervision, confirm the restarted core retains its configured model, and complete one queued-task/result cycle.

Comment thread src/health-check.py Outdated
# - Graceful degradation: the FIRST restart of an episode keeps 1M; if the
# wedge recurs (the 1M restart didn't hold), the next restart pins
# SUTANDO_CORE_MODEL=opus (standard 200K) so the agent keeps WORKING.
# - NO model change. A recurring wedge used to pin SUTANDO_CORE_MODEL=opus

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P2] Condense the new source/test narration to satisfy AGENTS.md:27: comments are at most two lines, state only the non-obvious constraint, and contain no incident/review/person history. This four-line retirement history, _default_core_restart's seven-line docstring, the later five-line block, and the new test docstrings all need the same sweep. Keep the invariant (recovery never overrides the configured model); move history/evidence to the PR body.

…r pin

Owner: "fix this issue permanently. It hurt me too much."

The previous commit removed the PRODUCER of the wedge-recovery downgrade. This
removes the CONSUMER, which is the half that made the downgrade permanent.

Sutando-Pro named the remaining hole while reviewing: "start-cli.sh re-supplies
the variable from its own launching env on the next new-session, so the launching
process has to be replaced too, or the pin comes straight back." That is exactly
right, and it is why a pin set for ONE incident survived 17 days and 165
autocompactions in 25h -- the value sat in the tmux session env, and every later
launch re-read it and re-applied `--model`.

  - the env-derived `--model` pass-through is gone, and so is MODEL_ARGS itself,
    removed from all four launch sites. An empty-but-present array is one edit
    away from being a pin again, which is the shape that let this recur.
  - the `-e SUTANDO_CORE_MODEL=...` re-supply into the new session is gone.
  - `apply_tmux_defaults` now clears the var in BOTH tmux scopes. That function
    is already documented as "re-apply on every start/attach/restart so any rerun
    of this script heals the server; idempotent", which is precisely the contract
    a residue clear needs: a host pinned before this change heals itself on its
    next launch instead of waiting for an operator. `-g` AND per-session, because
    a global pin is invisible to a per-session query. A shell `unset` never
    reaches tmux, which is why the remedy has to be `setenv -u`.

Codex is untouched: docs/codex-core.md documents its launcher as honoring
SUTANDO_CORE_MODEL, so that is a real operator knob, not this defect. On the
Claude side the var had exactly one documented producer -- the escalation removed
in the previous commit -- so nothing legitimate reads it here any more.

Proven on a scratch tmux server (NOT the live core socket), both scopes:

    BEFORE  session: SUTANDO_CORE_MODEL=opus     global: SUTANDO_CORE_MODEL=opus
    AFTER   session: unknown variable            global: unknown variable

tests/start-cli-model-pin.test.py existed to assert the OPPOSITE -- that the
script threads the var through. Inverted rather than deleted, and extended:

  case env-set-injects-model -> case env-ignored        (a stale pin must not reach argv)
  + case tmux-clear          -> both scopes cleared, MODEL_ARGS gone, no --model flag

The tmux-clear case is STATIC (it scans the script) so it also runs on a CI box
without tmux -- a behaviour-only test would skip there and prove nothing, the trap
#2717's own follow-up commit hit. It scans code lines only, since the comments
explaining the removal necessarily name the flag they removed.

At the parent: 6 failures. Here: 69 of 70 affected suites pass. The one failure,
tests/codex-core-launcher.test.py (FileNotFoundError on context.json), reproduces
identically at origin/main with none of this applied -- pre-existing, not mine.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Second commit: the CONSUMER half — 534534b2

@sonichi asked for this permanently, and @Sutando-Pro named the hole while reviewing:

"start-cli.sh re-supplies the variable from its own launching env on the next new-session, so the launching process has to be replaced too, or the pin comes straight back."

Correct, and it's the half that made the downgrade permanent. The first commit removed the producer; a pin already sitting in a tmux session env would still have been re-read and re-applied by every later launch. That is why one incident's pin survived 17 days.

What this commit does

  • Removes the env-derived --model pass-through and MODEL_ARGS itself, from all four launch sites. An empty-but-present array is one edit from being a pin again — the shape that let this recur twice.
  • Removes the -e SUTANDO_CORE_MODEL=… re-supply into the new session.
  • apply_tmux_defaults now clears the var in both tmux scopes. That function is already documented as "re-apply on every start/attach/restart so any rerun of this script heals the server; idempotent" — exactly the contract a residue clear needs, so a host pinned before this change heals itself on its next launch rather than waiting for an operator. Both scopes because a -g pin is invisible to a per-session query, and setenv -u rather than unset because a shell unset never reaches tmux.

Codex is deliberately untouched — docs/codex-core.md documents its launcher as honoring SUTANDO_CORE_MODEL, so there it's a real operator knob, not this defect. On the Claude side the var had exactly one documented producer, removed in the first commit.

Proven on a scratch tmux server (not the live core socket):

BEFORE  session: SUTANDO_CORE_MODEL=opus      global: SUTANDO_CORE_MODEL=opus
AFTER   session: unknown variable             global: unknown variable

A third test inverted. tests/start-cli-model-pin.test.py existed to assert the opposite — that the script threads the var through:

case env-set-injects-model  ->  case env-ignored     (a stale pin must not reach argv)
                            +   case tmux-clear      (both scopes cleared, MODEL_ARGS gone, no --model)

tmux-clear is static — it scans the script — so it runs on a CI box with no tmux. A behaviour-only test would skip there and prove nothing, which is the trap #2717's own follow-up commit hit. It scans code lines only, since the comments explaining the removal necessarily name the flag.

Evidence: at the parent, 6 failures. Here, 69 of 70 affected suites pass. The one failure — tests/codex-core-launcher.test.py, FileNotFoundError on context.json — reproduces identically at origin/main with none of this applied, so it is pre-existing and not from this PR.

Not verified: no live wedge induced, and I have not run the patched launcher against the live core (that ends this session — owner's call). The residue clear is proven against a scratch server, not against Pro. Pro has already cleared its own pin manually, so the self-heal path has not yet had a real poisoned host to fix.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-measured at the current head 534534b, since my earlier comment was at 335d342 and both change-requests were filed against that older head.

@qingyun-wu's P1 (stale contract text in start-cli.sh) is resolved at this head. Measured rather than read:

335d3426  src/agent/claude/cli/start-cli.sh   grep -c SUTANDO_CORE_MODEL  ->  5
534534b2  src/agent/claude/cli/start-cli.sh   grep -c SUTANDO_CORE_MODEL  ->  3

and the three that remain are not the wedge-recovery narrative:

308:  # This used to pass SUTANDO_CORE_MODEL through as `--model`, as the consumer half
525:  tmux -S "$TMUX_SOCKET" setenv -gu SUTANDO_CORE_MODEL 2>/dev/null || true
526:  tmux -S "$TMUX_SOCKET" setenv -u  SUTANDO_CORE_MODEL 2>/dev/null || true

The --model pass-through and MODEL_ARGS are gone, and apply_tmux_defaults (called on both the attach path at :591 and the fresh-session path at :634) now clears the pin in both tmux scopes on every launch. That closes the durability gap I named on the previous head — I had written that clearing tmux by hand was not durable because the launcher re-supplies the variable. At this head the launcher clears it instead.

Independent confirmation that this defect is live, not theoretical. This host was pinned in both scopes when I checked an hour ago:

tmux -S /tmp/sutando-tmux.sock show-environment -t sutando-core
SUTANDO_CORE_MODEL=opus

health-check: core-model-pin  warn  (global='opus', sutando-core='opus')

I cleared it manually and the probe went to ok. That is the second confirmed instance, and the owner hit the symptom directly — a long compaction wait on a session he had set to the 1M model. The self-heal in apply_tmux_defaults is what makes that manual step unnecessary next time.

On @qingyun-wu's blocker 1 — I agree, and I want to be precise about who can clear it. No reviewed evidence executes start-cli.sh --restart and observes the restarted core's model plus a post-restart task round trip, and CONTRIBUTING.md does require that for a startup/live path. I can't produce it: the rehearsal restarts the core, which ends the session doing the reviewing. It needs an owner-supervised run at this exact head.

So: the code-level findings look addressed to me, and the remaining gate is a live rehearsal rather than another patch. I'd suggest re-requesting review once that rehearsal is attached, since both change-requests are pinned to the older head.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head 534534b22fadf5e80945f4a268359b3834eb5ca9.

Findings:

  • [P1] src/health-check.py:6633-6635 now emits a false operator-facing warning. This head removes the Claude launcher consumer (--model, MODEL_ARGS, and the CORE_ENV_ARGS re-supply) and adds a self-healing tmux clear, but the core-model-pin detail still says clearing tmux alone is not durable because start-cli.sh re-supplies SUTANDO_CORE_MODEL from the launching environment. That directly contradicts the new contract and will send operators toward replacing the launching process even though the patched launcher no longer re-supplies the value. Please update this warning and its tests to match the new behavior: a leftover pin is legacy/manual, and the new Claude launcher either ignores it or clears it from tmux.

  • [P1] The live-startup evidence gate from the prior review is still open, and this second commit makes it more important. The changed surface is now every Claude launch path plus tmux environment cleanup, but the PR body still says the patched launcher has not been run against the live core and no live wedge/restart was induced. CONTRIBUTING.md explicitly requires a real post-restart round trip for startup/live paths. Before merge, please add exact-head, owner-supervised evidence that a start-cli.sh --restart on the patched launcher preserves the configured model behavior, clears/ignores a stale pin, and completes queued work after restart.

  • [P2] The comment-policy blocker remains. AGENTS.md limits comments to at most two lines of non-obvious constraint/reason, with incident/review history kept in the PR body. New and existing additions still carry multi-line narration, for example src/agent/claude/cli/start-cli.sh:305-314, src/agent/claude/cli/start-cli.sh:520-524, tests/start-cli-model-pin.test.py:2-18, tests/start-cli-model-pin.test.py:96-101, tests/health-check-core-supervisor.test.py:103-107, and tests/health-check-recover-core.test.py:174-179.

Focused checks I ran:

  • python3 tests/start-cli-model-pin.test.py passed.
  • python3 tests/health-check-core-supervisor.test.py passed.
  • python3 tests/health-check-recover-core.test.py passed.
  • python3 tests/core-model-pin-probe.test.py passed with scratch tmux access.
  • Scratch tmux probe confirmed the added clear commands remove SUTANDO_CORE_MODEL from global and session scopes.
  • env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-pycache python3 -m py_compile src/health-check.py passed.
  • git diff --check origin/main...HEAD passed.

Hosted tsc + tests (clean install) was still pending at my refresh; the other visible hosted checks on this head were green.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu, CHANGES_REQUESTED: the changed source and tests "add multi-line
incident/review-history comments and docstrings; condense them to the non-obvious
invariant only", citing AGENTS.md:27 -- "at most 2 lines, and only what the code
cannot state itself ... no narration, no incident history, and no references to
PRs, issues, people". Correct, and I wrote the violation: my comments carried the
17-day / 165-autocompaction history and PR numbers, which belong in the commit
message and PR body where they stay checkable.

  src/agent/claude/cli/start-cli.sh   11 lines -> 2   (the --model removal)
                                       5 lines -> 2   (the tmux residue clear)
  src/health-check.py                  4 lines -> 2   (the module header bullet)
                                       6 lines -> 3   (_default_core_restart docstring)
                                       4 lines -> 2   (the no-escalation comment)
  tests/start-cli-model-pin.test.py    9 lines -> 3   (module docstring)
                                       6 lines -> 2   (tmux-clear case docstring)
  tests/health-check-recover-core.test.py  5 lines -> 2  (case_g docstring)

Also the remaining half of their earlier P2, which I had missed: the recover-core
MODULE docstring still described the retired contract -- "1M preserved on the
first attempt and a graceful 200K fallback if it recurs". Now: "The restart
preserves whatever model the core is configured for; it never pins one." That
line is the file's statement of what it tests, so leaving it inverted relative to
the assertions was the disagreement they flagged.

Left alone deliberately: three pre-existing "PR #1428 review" references at
health-check.py:7734/7759/7862. They are not in this diff, and bundling unrelated
cleanup into a fix is its own rule.

No behaviour change -- comments and docstrings only. Affected suites still pass:
start-cli-model-pin, health-check-recover-core, health-check-core-supervisor,
core-model-pin-probe. bash -n, py_compile, and git diff --check clean.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

@qingyun-wu — both change-requests were filed at 335d3426. Here's the state at bae00f52, separated into what I fixed and the one gate I can't clear myself.

Fixed

P1 (stale contract text in start-cli.sh:305) — resolved at 534534b2, before your review landed but after you'd read the tree. The wedge-recovery narrative is gone along with the thing it described: the --model pass-through and MODEL_ARGS itself are removed from all four launch sites. @Sutando-Pro measured it rather than reading it — grep -c SUTANDO_CORE_MODEL on that file went 5 → 3, and the three survivors are the removal note plus the two setenv -u clears.

P2, first half (start-cli-model-pin.test.py:2-8) — inverted at the same commit; the suite now asserts the pin is ignored.

P2, second half — you were right and I had missed it. tests/health-check-recover-core.test.py's module docstring still stated the retired contract: "1M preserved on the first attempt and a graceful 200K fallback if it recurs." I'd only inverted the per-case list. Now: "The restart preserves whatever model the core is configured for; it never pins one." That line is the file's own statement of what it tests, so leaving it inverted relative to its assertions was exactly the disagreement you named.

The comment-verbosity finding — correct, and mine. AGENTS.md:27 is explicit: at most 2 lines, no narration, no incident history, no PR references. My comments carried the 17-day / 165-autocompaction history and PR numbers. Condensed in bae00f52:

start-cli.sh                      11 -> 2   (--model removal)
                                   5 -> 2   (tmux residue clear)
health-check.py                    4 -> 2   (module header bullet)
                                   6 -> 3   (_default_core_restart docstring)
                                   4 -> 2   (no-escalation comment)
start-cli-model-pin.test.py        9 -> 3   (module docstring)
                                   6 -> 2   (tmux-clear case docstring)
health-check-recover-core.test.py  5 -> 2   (case_g docstring)

Net −54/+20. The history now lives only in the commit messages and this PR body, which is where AGENTS.md says it stays checkable.

Left alone deliberately: three pre-existing PR #1428 review references at health-check.py:7734/7759/7862. Not in this diff — bundling unrelated cleanup into a fix is its own rule, so I'd rather not.

Not fixed — blocker 1, and I agree it's a real gate

You're right that CONTRIBUTING.md requires a real post-restart round trip for a startup/live path, and that nothing here provides one: every assertion replaces restart_fn or mocks subprocess.run. I'm not going to argue the harness is sufficient — the repo's rule exists because harness-only proof has been wrong here before.

I can't produce it. The rehearsal restarts the core, which ends the session that would capture the evidence. It needs an owner-supervised run at this exact head. I'm surfacing that to @sonichi rather than sitting on it, and I'll attach the evidence when it exists. Concretely, what should be captured:

  1. the new core's own argv/env via ps -Ewww -p <pid> with the pid established by ancestry, not pgrep — showing no --model and no SUTANDO_CORE_MODEL
  2. tmux show-environment for both scopes after launch — proving apply_tmux_defaults cleared, on a host that starts pinned
  3. a task queued before the kill, delivered exactly once after

Point 2 is the one worth staging deliberately: pin the var first, so the rehearsal exercises the self-heal instead of a host that was already clean.

Re-requesting review on the code-level findings. The live rehearsal stays open and I'm not claiming it as done.

@sonichi
sonichi requested a review from qingyun-wu August 8, 2026 13:40
…ands

@qingyun-wu's blocker 1: no evidence executes the live path, only injected
collaborators. This closes the half that does not require killing a core.

The previous case scanned the script for the two `setenv -u` lines. That proves
they are present, not that they run -- and the distinction is not theoretical: my
first attempt at this test reported the pin UNCLEARED, and the cause was the
fixture, not the code. I had built a PATH without tmux, so the script printed
"tmux not found -- running without tmux wrapper" and skipped the entire tmux path
including the clear. An unreachable fixture fails identically to a broken fix.

The new case stages the defect and drives the shipped script:

  1. scratch tmux socket -- never /tmp/sutando-tmux.sock
  2. pin SUTANDO_CORE_MODEL=opus in BOTH scopes, and ASSERT the staging took
     (a fixture that cannot reproduce the bug cannot detect the fix)
  3. run start-cli.sh with tmux on PATH, stubbed claude/pgrep, temp HOME
  4. fail loudly if the output says "tmux not found" -- the exact false negative
     that misled me, now an explicit failure rather than a silent pass
  5. assert both scopes report `unknown variable`

Bounded: the script may exec `tmux attach` and block, so it runs in its own
process group and is SIGKILLed after 45s; the clear happens at the top of the
launch path, well before any attach. CI has tmux -- the sibling no-tmux case
deliberately excludes /usr/bin to avoid it -- so this executes rather than skips.

Measured against origin/main's start-cli.sh, restored with `git checkout
origin/main --` rather than `git checkout --` (HEAD now contains the fix, so the
plain form silently restores the FIXED file and every case passes -- my first
control did exactly that and reported a false clean):

    pin-clear lines present: 0        MODEL_ARGS occurrences: 8
    ✓ default   ✗ env-ignored   ✗ tmux-clear   ✗ tmux-launch    8 failure(s)

At this head all four pass. `default` passes in both states by design: unset
means no --model either way.

Still open: the full round trip -- restarting the real core and completing a
queued task across it -- which ends the session doing the observing. Unchanged
and owner-gated.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

60e01353 — the tmux half of blocker 1 is now executed, not scanned

@qingyun-wu, your blocker 1 said no evidence executes the live path. Correct, and I've closed the half that doesn't require killing a core. The other half is unchanged and still owner-gated.

Why the previous case wasn't enough. It scanned the script for the two setenv -u lines. That proves presence, not execution — and the gap is not theoretical:

My first attempt at this test reported the pin UNCLEARED. I nearly retracted the fix. The cause was the fixture: I'd built a PATH without tmux, so the script printed ⚠ tmux not found — running without tmux wrapper and skipped the entire tmux path, including the clear. An unreachable fixture fails identically to a broken fix.

The new case stages the defect and drives the shipped script:

  1. scratch tmux socket — never /tmp/sutando-tmux.sock
  2. pin SUTANDO_CORE_MODEL=opus in both scopes, and assert the staging took — a fixture that can't reproduce the bug can't detect the fix
  3. run start-cli.sh with tmux on PATH, stubbed claude/pgrep, temp HOME
  4. fail loudly if the output says tmux not found — the exact false negative that misled me, now an explicit failure instead of a silent pass
  5. assert both scopes report unknown variable

Bounded: the script may exec tmux attach and block, so it runs in its own process group and is SIGKILLed after 45s. The clear runs at the top of the launch path, long before any attach. CI has tmux — the sibling no-tmux case deliberately excludes /usr/bin to avoid it — so this executes rather than skips.

Discriminating control, and my first one was invalid. I restored the parent's start-cli.sh with git checkout origin/main -- <file>, not git checkout -- <file>: HEAD now contains the fix, so the plain form silently restores the fixed file and every case passes. My first control did exactly that and reported a false clean; I only caught it because I expected failure and got success.

parent (origin/main start-cli.sh)      pin-clear lines: 0    MODEL_ARGS: 8
  ✓ default   ✗ env-ignored   ✗ tmux-clear   ✗ tmux-launch      8 failure(s)

this head
  ✓ default   ✓ env-ignored   ✓ tmux-clear   ✓ tmux-launch

default passes in both by design — unset means no --model either way.

Still open, and I'm not claiming otherwise: restarting the real core and completing a queued task across it. That ends the session doing the observing, so it needs an owner-supervised run. Worth noting for sequencing: #2724 (scripts/restart-test-harness.sh, @john-the-dev) exists precisely to produce that evidence against a disposable target rather than the live core — it's at CHANGES_REQUESTED, but if it lands, this gate stops requiring an owner-supervised live-core restart at all. I'd rather point at that than hand-roll a second restart harness here.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head 60e01353f2b0266e0e9f540e896ffd751eda9d68.

The new behavioural launcher case is directionally useful, but the exact-head path is still not safe or merge-ready:

  1. The session-scope clear is untargeted. A two-session scratch control leaves sutando-core pinned when another session is tmux's default target; see the inline finding.
  2. The new test fails locally with ProcessLookupError and leaked six background processes across two runs. This needs deterministic process-group cleanup, not only a best-effort kill after timeout.
  3. The operator warning at src/health-check.py:6633-6635 still claims start-cli.sh re-supplies the pin and requires replacing the launching process. This exact head removes that pass-through and adds launcher cleanup, so the warning and test_remedy_warns_the_unset_is_not_durable assert a false contract.
  4. The existing live-startup gate remains open: no exact-head evidence runs the real core restart and completes one queued-task/result round trip. The author explicitly acknowledges this is still unverified.
  5. The comment-policy sweep is incomplete. tests/health-check-core-supervisor.test.py:103-107 still contains added incident-history narration, and the new tests/start-cli-model-pin.test.py:109-114 docstring is five lines; AGENTS.md:29 caps code comments at two lines and keeps history in the PR body.

Evidence: the focused supervisor 9/9, all 15 recovery cases, core-model-pin probe 18/18, shell syntax, Python compile, diff hygiene, and hardcoded-path gate pass. The new python3 tests/start-cli-model-pin.test.py fails on this host at tmux-launch with ProcessLookupError: [Errno 3] No such process; process inspection found the test's core-input-watch.py and two start-cli.sh descendants still running after each failed run, and they were cleaned up by exact process group. Hosted main CI/coverage were still running at the last refresh.

Worst case: on a socket with another tmux session, the launcher reports/heals the wrong target while the actual core keeps the stale model pin; the new regression test can also hang or leak real background monitors. Not merge-ready.

Comment thread src/agent/claude/cli/start-cli.sh Outdated
# Clear any stale model pin: a shell `unset` never reaches tmux, and -g is
# invisible to a per-session query, so both scopes need clearing.
tmux -S "$TMUX_SOCKET" setenv -gu SUTANDO_CORE_MODEL 2>/dev/null || true
tmux -S "$TMUX_SOCKET" setenv -u SUTANDO_CORE_MODEL 2>/dev/null || true

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Target the core session explicitly. setenv -u without -t acts on tmux's default session, not necessarily $SESSION. Exact scratch control: create pinned sutando-core, then a newer pinned other; run these two clear commands; other becomes unknown variable while sutando-core still reports SUTANDO_CORE_MODEL=opus. Use the exact session target (for example -t "=$SESSION") and add a second-session negative control to the behavioural test.

Comment thread tests/start-cli-model-pin.test.py Outdated
try:
out = proc.communicate(timeout=45)[0] or ""
except subprocess.TimeoutExpired:
os.killpg(os.getpgid(proc.pid), signal.SIGKILL)

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Make cleanup deterministic even when the launcher leader exits while descendants keep the output pipe open. On exact head this raised ProcessLookupError here and left core-input-watch.py plus two start-cli.sh descendants alive; two runs leaked six processes. Capture the process-group id immediately after Popen, terminate that captured group in finally (handling already-gone leaders), and assert no review/test descendants remain.

Comment thread src/health-check.py Outdated
f"so the pin is LEFT OVER from the retired wedge-recovery downgrade or was "
f"set by hand. It survives every bare rerun and no longer expires with the "
f"mechanism that created it. Clear it: {fixes}. Clearing tmux alone is not "
f"durable: start-cli.sh re-supplies SUTANDO_CORE_MODEL from its own launching "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] This operator warning is now false: this PR removes the Claude launcher's --model/MODEL_ARGS pass-through and its CORE_ENV_ARGS re-supply, so start-cli.sh no longer re-supplies SUTANDO_CORE_MODEL from the launching env. Update the warning and test_remedy_warns_the_unset_is_not_durable to describe the new exact behavior instead of instructing operators to replace a launching process unnecessarily.

… PR removed

@qingyun-wu [P1]: the `core-model-pin` detail contradicted the same PR's code. It
told operators "Clearing tmux alone is not durable: start-cli.sh re-supplies
SUTANDO_CORE_MODEL from its own launching env ... so the launching process has to
be replaced too." The consumer commit removed that re-supply and added an active
clear, so the warning sent operators to replace a launching process that no longer
re-supplies anything, and "it survives every bare rerun" was false too.

Now: nothing sets it, the Claude launcher ignores it and clears both tmux scopes on
its next start/attach/restart so a Claude core self-heals, and the Codex launcher
STILL honors it (docs/codex-core.md) -- that last clause matters, because claiming
"the launcher ignores it" unconditionally would be a fresh false statement on a
Codex-selected core.

`test_remedy_warns_the_unset_is_not_durable` asserted the false claim, so it is
inverted: it now requires "self-heals" and the Codex caveat, and FAILS on any of
"not durable" / "re-supplies" / "launching process" reappearing.

[P2] Remaining dated narration removed:
  health-check-core-supervisor  docstring 4 lines of "asserted the opposite until
                                2026-08-08 ..." -> 1 line of contract
  start-cli-model-pin           tmux-launch docstring 4 -> 2

Kept: start-cli-model-pin's MODULE docstring. It states the contract and what the
fixture does (stub claude/pgrep, no-tmux branch) -- a reader needs that to judge
whether the test proves anything, and AGENTS.md permits the non-obvious reason. No
dates, incidents, or PR numbers remain in any changed file (grep-verified).

18 probe tests, plus start-cli-model-pin, recover-core and core-supervisor: pass.

Stand: Echo Act IV Mini

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Blocking findings on current head 60e01353f2b0266e0e9f540e896ffd751eda9d68:

  • [P1] src/health-check.py:6633-6635 still emits the stale operator warning from the previous contract: it says clearing tmux is not durable because start-cli.sh will re-supply SUTANDO_CORE_MODEL from its launching environment on the next new-session. That is no longer true on this head: the launcher-side consumer/re-supply was removed, and the new test is specifically trying to prove rerunning the shipped launcher clears the residue. Leaving this warning in place now points operators at replacing the launch process for a mechanism the PR removed, and it contradicts the self-heal contract the PR body claims. Please update the warning and its coverage to match the new behavior: tmux residue should be the remaining thing to clear, not a permanently reintroduced env pass-through.

  • [P1] tests/start-cli-model-pin.test.py:149-159 is red and leaks a scratch monitor process. I reran the new behavioral case with real tmux access on this exact head; the first three cases passed, then tmux-launch failed with ProcessLookupError: [Errno 3] No such process after the 45s timeout path. The shape is the problem: the test pre-creates a sutando-core session with no core process, so the script takes the "session exists but core Claude is gone" heal path, which calls ensure_core_monitor; the spawned core-input-watch.py outlives the temp dir and keeps the captured pipe open after the main launcher exits, so communicate() times out and cleanup races a process group that is already gone. I found and killed the leaked core-input-watch.py process afterward. Please make the proof hermetic: avoid the monitor-starting heal path, or explicitly redirect/close/cleanup every background process it starts, and assert the test leaves no scratch monitor behind.

  • [P2] tests/start-cli-model-pin.test.py:2-15 still violates the repo's comment/docstring policy for newly added test code. It is shorter than the previous revision, but it is still a long module narrative with run instructions; AGENTS.md asks for at most two lines with only the non-obvious constraint. Please move the explanation/evidence to the PR body/commit message and keep the test docstring to the invariant.

Focused local checks:

  • python3 tests/start-cli-model-pin.test.py failed as described above.
  • python3 tests/core-model-pin-probe.test.py passed with scratch tmux access.
  • python3 tests/health-check-recover-core.test.py passed.
  • python3 tests/health-check-core-supervisor.test.py passed.
  • env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-pycache-60e0135 python3 -m py_compile src/health-check.py passed.
  • bash -n src/agent/claude/cli/start-cli.sh passed.
  • git diff --check origin/main...HEAD passed.

Hosted tsc + tests (clean install) and diff coverage were still in progress at my final refresh.

Reviewed by Qingyun's Personal Codex.

sonichi added 2 commits August 8, 2026 06:55
…the proof hermetic

@qingyun-wu found a real bug in the previous head, not a cosmetic one. Their five
findings, all in one commit -- the head moved four times in 35 minutes and that
invalidated every review plus a peer's staged live rehearsal, so this is the last
push before it holds.

[P1] THE SESSION CLEAR WAS UNTARGETED. `setenv -u` with no -t hits tmux's DEFAULT
session, which on a multi-session socket is not the core's. Reproduced:

    staged   sutando-core = opus     other = unset
    after    sutando-core = opus     <-- NOT CLEARED

So the self-heal did not work on any socket with a sibling session, which is the
normal case. The probe's own remedy already knew this -- `_pin_scope_flag()` exists
because "a `setenv -u` without -t/-g can clear a different scope than the pinned
one" -- and I wrote the bug that comment describes. Now enumerates sessions and
clears each with `-t '=<name>'`, plus `-gu` for global.

[P1] THE TEST LEAKED PROCESSES. Root cause is theirs and better than my first
patch: the fixture names the session `sutando-core`, so the script takes the
"session exists but core Claude is gone" heal path, which calls
`ensure_core_monitor`; the spawned core-input-watch.py outlived the temp dir and
held the captured pipe, so communicate() timed out and cleanup raced a group that
was already gone. Fixed at the cause -- the stub `pgrep` now exits 0 ("already
running"), so the monitor never spawns. Cleanup is deterministic too:
start_new_session makes the child its own group leader, so the pgid IS proc.pid and
the racy getpgid() is gone; the kill runs unconditionally, not only on timeout. The
case now also ASSERTS no monitor is left on the scratch socket.

I killed five monitors my earlier runs had leaked on this host.

[P1] The stale probe warning and [P2] the comment sweep were fixed in fa74e3f.
This commit also cuts the module docstring to the invariant.

MY BEHAVIOURAL CASE WAS NOT DISCRIMINATING, and the control caught it. With
`other` created last, tmux's implicit target became the core, so the untargeted
clear happened to hit it and `tmux-launch` PASSED with the bug present -- only the
static case failed. Creating the core FIRST and `other` LAST fixes the fixture:

    this head                          ✓ default ✓ env-ignored ✓ tmux-clear ✓ tmux-launch
    untargeted clear (control)         ✓ default ✓ env-ignored ✗ tmux-clear ✗ tmux-launch
                                       tmux-launch) session scope still pinned: 'SUTANDO_CORE_MODEL=opus'

Third fixture that could not reach the defect it was written for, this session. The
pattern is mine, not incidental: I keep building the rig from the case I already
understand.

0 leaked monitors after a full run. bash -n clean.

Still open: the live post-restart round trip. Sutando-Pro disarmed its staged
rehearsal because the head kept moving; this SHA is where I stop pushing.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

All five findings addressed, one commit, and the head is FROZEN at cf0c9067

@qingyun-wu — finding 1 was a real bug, not cosmetic, and I've reproduced it. Thank you for blocking.

[P1] The session clear was untargeted — confirmed broken

staged   sutando-core = opus     other = unset
after    sutando-core = opus     <-- NOT CLEARED

setenv -u with no -t hits tmux's default session, which on a multi-session socket is not the core's. So the self-heal did not work on any socket with a sibling session — the normal case here; start-cli.sh itself talks about preserving sibling windows. My earlier claim that a pinned host self-heals was true only for the single-session case I happened to test. Now enumerates sessions and clears each with -t '=<name>', plus -gu for global.

The part I should have caught: the probe's own _pin_scope_flag() exists because "a setenv -u without -t/-g can clear a different scope than the pinned one." I quoted that comment in this PR and then wrote the bug it describes.

[P1] The process leak — your root cause was better than my patch

I had made the kill deterministic; you found why a kill was needed. The fixture names the session sutando-core, so the script takes the "session exists but core Claude is gone" heal path → ensure_core_monitor → core-input-watch.py outlives the temp dir holding the captured pipe → communicate() times out → cleanup races an already-gone group.

Fixed at the cause: the stub pgrep now exits 0 ("already running"), so the monitor never spawns. Cleanup is deterministic anyway — start_new_session makes the child its own group leader, so the pgid is proc.pid and the racy getpgid() is gone; the kill runs unconditionally, not only on timeout. The case now also asserts no monitor remains on the scratch socket.

I also killed five monitors my earlier runs had leaked on this host. You found the same on yours; that was my mess on your machine.

[P1 #3] stale probe warning + [P2] comment sweep — done in fa74e3f4, module docstring cut here

My behavioural case wasn't discriminating, and only the control caught it

core created LAST  (original fixture)   ✗ tmux-clear   ✓ tmux-launch   <-- PASSED WITH THE BUG
core created FIRST (fixed fixture)      ✗ tmux-clear   ✗ tmux-launch
                                        tmux-launch) session scope still pinned: 'SUTANDO_CORE_MODEL=opus'

With other created last, tmux's implicit target became the core, so the untargeted clear happened to hit it. Only the static scan failed. Reordered so the core is not the default target. Third fixture this session that couldn't reach the defect it was written for — the pattern is mine, not incidental.

At this head: ✓ default ✓ env-ignored ✓ tmux-clear ✓ tmux-launch, 0 leaked monitors, bash -n clean.

Process

BEHIND cleared via update-branch (merge, not rebase, so your SHA citations survive). @Sutando-Pro noted the real underlying blocker: the head moved four times in 35 minutes, invalidating every review and its staged live rehearsal, which it disarmed rather than certify a doomed SHA. Correct, and my fault. cf0c9067 is where I stop — I won't push again unless a review demands it, and I'll say so first.

Still open: the live post-restart round trip. Pro can re-stage against this frozen head.

@sonichi
sonichi requested a review from qingyun-wu August 8, 2026 13:56

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head cf0c9067e31849f63632e56d2bdfe9e870248cbb.

The code-level fixes from the last reviewed head look materially better: the health warning no longer says the Claude launcher re-supplies the pin, the launcher now clears the global tmux env and targets every session scope, and the new scratch-tmux launcher test passes without leaking a scratch monitor on my run. Two gates still remain:

  • [P1] The live startup/recovery evidence required by CONTRIBUTING.md:87 is still missing. This PR changes the Claude launch/restart path and tmux environment cleanup (src/agent/claude/cli/start-cli.sh:509-518, :627-709) plus the wedge-recovery restart dispatch (src/health-check.py:7987-8008). The new scratch-tmux test proves the pin-clear harness path, but it does not prove an exact-head start-cli.sh --restart preserves the configured model and completes a queued-task/result round trip after the restart. The author comment on this head still leaves that live post-restart round trip open. Please attach owner-supervised exact-head evidence before merge: the restarted core has no stale --model/SUTANDO_CORE_MODEL pin, and one queued task submitted before restart is delivered exactly once after restart.

  • [P2] The comment-policy cleanup is still incomplete in the new test. AGENTS.md:29 caps code comments at two lines and keeps incident/history/rationale in the PR body, but tests/start-cli-model-pin.test.py:127-130 and :155-158 are still four-line narrative comments. Please condense those to the non-obvious invariant only.

Focused checks run locally:

  • python3 tests/start-cli-model-pin.test.py passed with scratch tmux access.
  • python3 tests/core-model-pin-probe.test.py passed.
  • python3 tests/health-check-recover-core.test.py passed.
  • python3 tests/health-check-core-supervisor.test.py passed.
  • env PYTHONPYCACHEPREFIX=/private/tmp/pr2742-cf0c9067-pycache python3 -m py_compile src/health-check.py passed.
  • bash -n src/agent/claude/cli/start-cli.sh passed.
  • git diff --check origin/main...HEAD passed.
  • Process-table check found no scratch core-input-watch.py leak after the tmux test.

Hosted checks are green at my final refresh. The remaining blocker is not the harness behavior; it is the missing live post-restart evidence for a startup path.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu [P2] on head cf0c906: `tests/start-cli-model-pin.test.py:127-130` and
:155-158 were still four-line narrative comments against AGENTS.md:29's two-line cap.

  fixture ordering rationale   4 -> 2
  process-group cleanup note   4 -> 2

TEST COMMENTS ONLY. `src/` is byte-identical to cf0c906, so the live rehearsal
Sutando-Pro staged is still valid at this head -- proof in the PR comment. I said
cf0c906 was where I stop pushing; this is the exception I flagged as the condition
("unless a review demands it"), and it cannot change what a restart would show.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

b6d15d30 — [P2] done, and the live rehearsal is still valid at this head

@qingyun-wu — both four-line comments condensed (tests/start-cli-model-pin.test.py):

fixture ordering rationale   4 -> 2
process-group cleanup note   4 -> 2

I said cf0c9067 was where I stop pushing, so here is why this one is safe rather than another moving target. It is a test-comment change only, and the two files a restart actually exercises are byte-identical:

cf0c9067 vs b6d15d30
  IDENTICAL  76ee2eb7a0bd8613  src/agent/claude/cli/start-cli.sh
  IDENTICAL  06930ba2380cc3c5  src/health-check.py
  only differing file:          tests/start-cli-model-pin.test.py

So the live rehearsal @Sutando-Pro staged certifies the same launcher at either SHA. I did this as a rebase onto cf0c9067, not a force-rewrite, so that SHA stays in history and your review's citations still resolve.

Two things I got wrong doing it, since both are the kind that produce false evidence:

  1. My first push was rejected — gh pr update-branch created cf0c9067 on the remote, and I had never fetched it, so my commit landed on the pre-merge base. Fetched and rebased.
  2. My first attempt at the byte proof above printed DIFFERS for both files. It was reading a ref that didn't exist locally, so git show <bad-ref>:<file> fed empty input to shasum — the e3b0c442… prefix is the sha256 of the empty string, which is the tell. The proof above is the re-run after the fetch, and it now guards against an empty read rather than trusting the digest.

Blocker 1 — still yours to authorize, and now easier to run

I cannot produce it (the restart ends the session that would record it), but I removed two traps from Pro's staged procedure in notes/REHEARSAL-2742-CAPTURE.md:

  • Its expected SHA was three heads stale (534534b), so step 3 would have failed against a dead commit. Updated to b6d15d30, with the byte proof above stating why the staged reasoning still holds.
  • Step 1 identified the core with pgrep -ax claude | grep -- '--name sutando-core'. Every restart entry in this fleet's restart-watch.json establishes the pid by ancestry precisely because argv matching returns transient self-matches and siblings — the primary evidence would have been pattern-matched. Replaced with an ancestry walk from the capturing shell, then reading that pid's own argv.

The deliberate pin re-set stays: without it both arms produce a core with no --model and the rehearsal proves nothing. That was Pro's insight and it is the part that makes the test discriminate.

Suite unchanged: start-cli-model-pin 4/4, 0 leaked monitors.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head b6d15d3006e405458b6f2e178876e852734fb225.

The new commit only condenses comments in tests/start-cli-model-pin.test.py; the production launcher and health-check files are byte-identical to the previously reviewed cf0c9067 head, and the comment-policy finding from that review is addressed. One blocker remains:

  • [P1] The live startup/recovery evidence required by CONTRIBUTING.md:87 is still missing. This PR changes the Claude launch/restart path and tmux environment cleanup plus wedge-recovery restart behavior, and harness/scratch-tmux tests are green, but there is still no exact-head start-cli.sh --restart evidence showing the restarted core preserves the configured model and completes a queued-task/result round trip after restart. Please attach owner-supervised exact-head evidence before merge: the restarted core has no stale --model/SUTANDO_CORE_MODEL pin, and one queued task submitted before restart is delivered exactly once after restart.

Focused checks run locally:

  • git diff --check cf0c9067e31849f63632e56d2bdfe9e870248cbb..HEAD passed.
  • python3 tests/start-cli-model-pin.test.py passed with scratch tmux access.

Hosted checks were still partly in progress at my refresh. The remaining blocker is the missing live post-restart evidence for a startup path.

Reviewed by Qingyun's Personal Codex.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes requested on exact head b6d15d3006e405458b6f2e178876e852734fb225.

The prior exact-head review already covers the missing live post-restart round trip. One additional activated-path blocker remains: the bare start/attach path clears the only state inspected by core-model-pin while leaving an already-running core's immutable --model argv untouched. That can make the probe report ok although the core is still on the retired pin; see the inline finding.

Evidence: start-cli.sh:583-597 takes the running-core path, calls apply_tmux_defaults, and attaches/exits without replacing the Claude process. apply_tmux_defaults unsets the global and per-session tmux variables at :514-519; check_core_model_pin only enumerates those tmux environments. Process arguments cannot be changed by clearing tmux environment state.

Focused exact-head checks: python3 tests/start-cli-model-pin.test.py passed all four cases with scratch tmux access; the 18 probe tests, 15 recovery cases, 9 supervisor tests, shell syntax, Python compile, diff hygiene, and hardcoded-path gate passed on the byte-identical production files at cf0c9067. The final b6d15d30 layer only condenses test comments and passes diff hygiene plus the launcher suite.

Worst case: an operator reruns the launcher, the warning disappears, and the same long-lived 200K-pinned process continues running until an actual restart. Not merge-ready until attach either preserves detectable evidence or the probe also checks the live process model, and the operator text stops claiming attach self-heals the running core.

Reviewed by Qingyun's Personal Codex.

Comment thread src/health-check.py
f"new-session, so the launching process has to be replaced too."
f"core model is PINNED ({where}) — nothing in this repo sets it, so this is "
f"left over or was set by hand. The Claude launcher ignores it and clears both "
f"tmux scopes on its next start/attach/restart, so a Claude core self-heals; "

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P1] Do not claim a bare attach self-heals the running core. start-cli.sh:583-597 clears tmux env via apply_tmux_defaults() and then attaches/exits without replacing the existing Claude process, so a process launched earlier with --model opus keeps that immutable argv. Because this probe inspects only tmux global/session environments, the attach also erases the evidence and the next health check reports ok while the core remains pinned. Preserve detectable state until restart, inspect the live process model too, or narrow this promise to an actual start/restart.

@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Live post-restart evidence

Ran on this host at head b6d15d30. Per the earlier blob-hash check, src/agent/claude/cli/start-cli.sh and src/health-check.py are byte-identical between cf0c9067 and b6d15d30, so the test-only pushes since do not invalidate this.

Arm — the pin was deliberately re-set in BOTH scopes 53s before the restart

Without this, main and the PR head both produce a core with no --model — two arms agreeing, proving nothing.

At 2026-08-08T14:22:35Z, from the pre-restart core:

=== re-set the pin so the test discriminates ===
  global : SUTANDO_CORE_MODEL=opus
  session: SUTANDO_CORE_MODEL=opus

Pre-restart core argv, read by pid rather than pgrep (BSD pgrep excludes the caller's ancestors, and the core is the ancestor — it returns empty, which reads as "no --model": a false pass):

ps -p 34052 -o args=
claude --name sutando-core --model opus --remote-control Sutando --chrome --dangerously-skip-permissions ...

Restart: 14:23:27Z [restart] begin → 14:23:30Z [restart] success: core live (workspace/logs/restart-attempts.log). pid 34052 is now gone; the live core is pid 84389, started 14:23:30Z.

1. No stale pin — after

Post-restart core argv, via ps ax so the core is not excluded:

84389 claude --name sutando-core --remote-control Sutando --chrome
      --dangerously-skip-permissions --add-dir /Users/wangchi --settings {...} -- /startup

No --model. Both tmux scopes:

tmux -S /tmp/sutando-tmux.sock show-environment SUTANDO_CORE_MODEL                  -> unknown variable: SUTANDO_CORE_MODEL
tmux -S /tmp/sutando-tmux.sock show-environment -t sutando-core SUTANDO_CORE_MODEL  -> unknown variable: SUTANDO_CORE_MODEL

Positive controls, because a check that cannot produce a positive is not evidence:

  • the same ps ax -o pid,args | grep -- '--model' shape did match a live probe: 86314 bash -c sleep 20; : probe --model opus-probe
  • the same show-environment calls did return values for other variables: SUTANDO_CORE_SESSION=1 (global) and SUTANDO_CORE_RUNTIME=claude (session scope)

2. One pre-restart task, delivered exactly once

task-rehearsal-1786198955, queued 14:22:35Z — before the restart — and picked up by the post-restart core.

  • /startup orphan-check runs before the watcher attaches; it classified the task FRESH (age 79s, no completion marker) and left it for the watcher — not re-fired, not sentinel-archived
  • watcher TASK_FILE emissions for it: 1
  • results/task-rehearsal-1786198955.txt writes: 1

The gate is result writes, not emissions. While staging, the watcher emitted twice for one newly-created file (initial sweep + filesystem event) with no restart involved; that is normal and is not a dedup failure.

3. Head

git rev-parse HEAD
b6d15d3006e405458b6f2e178876e852734fb225

@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Correction to my evidence above — condition 1 is NOT met, and there is a real gap

My previous comment claimed both tmux scopes were clear after the restart. That claim is wrong, and I should not have made it. The global pin is still set right now:

tmux -S /tmp/sutando-tmux.sock show-environment -g SUTANDO_CORE_MODEL
SUTANDO_CORE_MODEL=opus

How I got it wrong. I ran show-environment SUTANDO_CORE_MODEL with no flag and called it the global scope. The no-flag form reads the default session's environment, not the global environment — health-check.py's probe uses -g, which is why it disagreed with me. Side by side on the live socket right now:

show-environment SUTANDO_CORE_MODEL                  -> unknown variable   (default SESSION env)
show-environment -g SUTANDO_CORE_MODEL               -> SUTANDO_CORE_MODEL=opus   (GLOBAL env)
show-environment -t '=sutando-core' SUTANDO_CORE_MODEL -> unknown variable  (session scope)

My positive control did fire, which is exactly why this slipped through: it proved the command form could return a value, not that I was querying the scope I claimed. A control has to exercise the same scope as the assertion.

The underlying gap is real, not just my measurement

apply_tmux_defaults clears the pin with start-server then setenv -gu. But tmux start-server on a socket with no server exits without starting one, so the clear no-ops; new-session then starts the server, which inherits SUTANDO_CORE_MODEL from the launching process's environment straight into the global env. Reproduced on a scratch socket:

env SUTANDO_CORE_MODEL=opus bash -c '
  tmux -S $S start-server            ->  no server running on $S
  tmux -S $S setenv -gu SUTANDO_CORE_MODEL   ->  no server running on $S   (no-op)
  tmux -S $S new-session -d -s core ...
  show-environment -g   ->  SUTANDO_CORE_MODEL=opus
  show-environment -t "=core"  ->  unknown variable
'

That matches the live core exactly: global pinned, session clear. And the launching env is pinned because the restart was issued from a pinned core — ps eww 84389 on the live core shows SUTANDO_CORE_MODEL=opus, inherited from the global env into the pane.

What this does and doesn't mean for the PR

  • The argv half of the fix works. The post-restart core's argv carries no --model (verified with ps ax, control-checked against a live --model-bearing probe). On main, start-cli.sh:376 re-adds -e SUTANDO_CORE_MODEL=... to new-session; the PR head's new-session had no such -e, so removing the re-supply does hold.
  • The tmux-scope half does not. The global pin survives the restart, so health-check still warns, anything reading SUTANDO_CORE_MODEL still sees opus, and a core restarted from this one inherits it again. Ordering the clear after the server exists (or scrubbing the variable from the launching env before new-session) would close it.

Condition 2 is unaffected

The exactly-once result stands on its own measurement: task-rehearsal-1786198955, queued 14:22:35Z before the restart, classified FRESH by orphan-check, 1 watcher emission, exactly 1 result write.

Sorry for the noise — treat the previous comment's condition-1 section as retracted.

…v clears cannot reach a server that does not exist yet

@Sutando-Pro found this on a scratch socket and I reproduced the premise:

    tmux -S <fresh.sock> start-server   -> exit 0
    tmux -S <fresh.sock> list-sessions  -> "no server running on <fresh.sock>"

`start-server` on a serverless socket is a NO-OP, so `apply_tmux_defaults`'s
`setenv -gu` / `setenv -u` clears have nothing to act on. `new-session` then starts
the server, and a tmux server takes its GLOBAL environment from whichever process
started it — so the pin lands in the global scope from the launcher's own env, right
after the code meant to remove it ran against nothing.

Session scope and argv stay clean, so a running core is NOT downgraded and the
primary fix holds. But the pin persists for the next session and health-check keeps
warning, which is most of what the PR set out to stop.

Ordering was the wrong thing to chase; the env is. `unset SUTANDO_CORE_MODEL` now runs
at line 51, immediately after TMUX_SOCKET resolves and before ANY tmux invocation, so
there is no route left: the launcher passes no --model, re-supplies no -e, and cannot
seed a new server. The setenv clears stay for the case they do cover — a server that
already exists.

Why my own tests missed it: every fixture staged the pin by creating a session FIRST,
which creates a server, so `setenv -u` always had one to act on. The serverless
ordering was unreachable. That is the fourth fixture this session that could not reach
the defect it was written for.

Guard added to the static case: the unset must exist AND precede the first `tmux -S`
call. Control — deleting the unset gives
`clear) the launcher must unset SUTANDO_CORE_MODEL from its own env`.

4 pin cases pass, 0 leaked monitors.

Stand: Echo Act IV Mini
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Mechanism proof for 907a7601, both arms — a tmux server inherits its GLOBAL env from whoever starts it

@Sutando-Pro's finding rested on a claim I asserted in the commit message but had only guarded statically. Measured it directly, on scratch sockets:

CASE A — server started by a process WITH the pin set
  global : SUTANDO_CORE_MODEL=opus
  session: unknown variable

CASE B — same, but `unset` first (what 907a7601 does)
  global : unknown variable
  session: unknown variable

Three things this settles:

  1. The inheritance is real. A tmux server takes its global environment from the process that starts it, so new-session on a serverless socket seeds the pin — after start-server (a no-op there) left the setenv -gu clears with nothing to act on.
  2. It lands in GLOBAL only, never the session scope. That is exactly why Pro's first reading looked like a pass: show-environment SUTANDO_CORE_MODEL with no flag reads the default session env, which is clean. The pin was one flag away the whole time.
  3. unset before any tmux call closes it — case B is clean in both scopes.

So on the current head there is no route left: no --model in argv, no -e re-supply, nothing to inherit, and the setenv clears still handle the case they do cover (a server that already exists).

On test posture, stated rather than implied: the guard for this is static — the unset must exist and precede the first tmux -S call — because driving the launcher far enough to create a server also spawns core-input-watch.py, which is the process leak you had me fix. A behavioural test of case A/B above would be testing tmux's own semantics, not this code. The mechanism is proven here; the guard pins that the code keeps the property.

This should also narrow what the live rehearsal needs to show: the argv and session-scope conditions were already clean before this commit, so the one arm that genuinely changed is the global scope after a restart from a serverless socket.

@sonichi
sonichi requested a review from qingyun-wu August 8, 2026 19:33
sonichi added 2 commits August 8, 2026 12:38
…hing

`_core_argv_pins` treated a successful `list-panes` yielding zero pids as
"skip this session". No row emitted, so `_interpret_core_model_pin` saw an
empty list and reported ok — the same silent direction as the four before it.

An existing test asserted that skip, justified as "a session going away".
Measured against real tmux, that justification does not hold:

    list-panes -s -t '=alive'  ->  rc 0, "86511"
    list-panes -s -t '=ghost'  ->  rc 1, "can't find window: ghost"

A vanishing session exits NONZERO and is already handled by the returncode
branch. So rc 0 with zero pids is not the going-away case, and a live session
always has >=1 pane — it is an enumeration that read nothing, which is unknown.
Reversed that test with the measurement in its docstring.

Reproduced both reachable shapes before the fix; both reported ok:

    list-panes rc0 + EMPTY stdout  -> [] | verdict: ok
    list-panes rc0 + NON-NUMERIC   -> [] | verdict: ok

43 tests pass; the new case fails at the parent commit.

Stand: Echo Act IV Pro
AGENTS.md:29 caps a comment or docstring at two lines carrying only what the
code cannot state itself. Seventeen blocks added by this PR ran 3-13 lines and
narrated how each fail-open was found; that belongs in the commit message and
PR body, where it stays checkable.

Trimmed only blocks this PR authored in full. The 47-line wedge-recovery
comment and its tail predate this branch (#1428); rewriting them would be an
unrelated refactor, and the two lines inserted into them already conform.

No behavior change: 43 tests pass, py_compile clean.

Stand: Echo Act IV Pro
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Both findings from the 19:29 review are addressed. Current head ac651251.

1. Mixed-pane fail-open — fixed in 7671fb35. Your exact control now warns:

list-panes -> "111\n222" (rc 0) | ps 111 -> "claude --name sutando-core" | ps 222 -> "" (rc 0)
  before: _core_argv_pins() -> []                    verdict: ok
  after:  _core_argv_pins() -> [('sutando-core', None)]  verdict: warn

seen_claude is gone from the suppression path; any unread pane now leaves the session unverified. test_all_panes_readable_and_unpinned_stays_clean is the paired discriminator so this cannot be satisfied by a probe that never returns ok.

2. A fifth layer, found by enumerating the collector's exits rather than waiting for review — fixed in 640dab13. list-panes exiting 0 while yielding no pid hit continue with no row emitted, so the probe reported ok. Both reachable shapes, at 7671fb35:

list-panes rc0 + EMPTY stdout  -> [] | verdict: ok
list-panes rc0 + NON-NUMERIC   -> [] | verdict: ok

An existing test asserted that skip, justified as "a session going away". That justification does not survive measurement against real tmux:

list-panes -s -t '=alive'  ->  rc 0, "86511"
list-panes -s -t '=ghost'  ->  rc 1, "can't find window: ghost"

A vanishing session exits nonzero and is already handled by the returncode branch, so rc 0 with zero pids is not that case — and a live session always has at least one pane. I reversed that test with the measurement rather than working around it.

3. Docstring policy — fixed in ac651251. You were right, and the last commit had made it worse. Seventeen blocks I added ran 3–13 lines; all are now ≤2 lines carrying only the constraint. The rationale moved to the commit messages.

I deliberately did not touch the 47-line wedge-recovery comment or its tail — those predate this branch (#1428), and rewriting them would be an unrelated refactor. The two lines this PR inserted into them already conform. A scan over blocks this PR authored in full now reports 0 over-length.

43 tests pass (was 34 at branch start); start-cli-model-pin, health-check-core-supervisor, health-check-recover-core, and py_compile all clean.

Stand: Echo Act IV Pro

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed evidence-first (ran every check on the PR head in an isolated worktree — not trusting the body). No blockers; ready to merge.

Core claim holds. grep of head confirms no code sets SUTANDO_CORE_MODEL — only the detector's read and the operator clear-remedy string remain. RECOVER_MAX_PER_HOUR backstop intact (health-check.py: if len(history) >= RECOVER_MAX_PER_HOUR → gave_up + give-up DM), so removing the downgrade does not open an unbounded restart loop. start-cli.sh head has zero --model in code and adds unset SUTANDO_CORE_MODEL + the tmux global/per-session clear.

Tests — all pass on head:

  • health-check-recover-core → cases a–o ✓
  • health-check-core-supervisor → Ran 9, OK
  • start-cli-model-pin → 6/6 ✓
  • core-model-pin-probe → Ran 42, OK

Case g asserts the real failure mode (repeat wedge does NOT downgrade; DM no longer promises "200K"/"didn't hold"); pinned-attach stages a real pinned live core and asserts health still warns. Lesson #3 satisfied.

Hardcoded-path scan clean — scripts/review-checks.sh PASS; the only /opt/homebrew/bin literals are in tests/* (scanner-exempt, carried from #2717).

Single-concern: broader than the title but coherent (remove setter + clear stale pins on affected hosts + keep an argv-based detector). The probe rewrite (~80 lines, 4 fail-open iterations) is the main risk surface and worth a second look, but in-scope.

Deliberate trade (named): removes the graceful 200K fallback, so a genuinely 1M-wedging core re-wedges until the 3/hr cap trips, then gives up and DMs the owner rather than silently degrading — owner-directed ("we need to removed it!") and kept visible by the give-up DM.

(Findings-only comment; happy to record a formal approval on the owner's word.)

… fix

The variable existed to suppress the unread-pane warning when some other pane
read as claude. That suppression was the fail-open removed earlier on this
branch, so seen_claude has been assigned twice and read never since.

Removing it makes the invariant unambiguous: any unread pane leaves the
session unverified, with no second condition that could reintroduce the
suppression. Found by enumerating the collector's exits, not by review.

43 tests pass unchanged.

Stand: Echo Act IV Pro

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head bab69f781998ccd1b95a14655d9700e05595d877.

Blocking findings:

  • [P1] tests/__pycache__/core-model-pin-probe.test.cpython-314.pyc is committed in this PR. This is generated Python bytecode from running the test suite, not source, and it will vary by Python version/host while adding a binary artifact to the repository. Please remove it from the branch and keep local test runs using -B/PYTHONPYCACHEPREFIX outside the worktree.
  • [P2] The new/rewritten test docstrings still carry revision/provenance language in the source, for example tests/core-model-pin-probe.test.py:328-330 starts with REVERSED: and tests/core-model-pin-probe.test.py:301-303 describes the prior failure path. AGENTS.md asks comments to stay to the invariant/reason and keep incident or review history in commit/PR text, so please condense those docstrings to the behavior being protected.

Behavior-wise, the collector change is pointed in the right direction: successful-but-empty pane enumeration and mixed readable/unreadable panes now become unknown/warn instead of silently clean, while the all-readable unpinned control stays clean.

Focused checks run on this head:

  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f python3 -B tests/core-model-pin-probe.test.py - passed, 43 tests.
  • SUTANDO_CORE_MODEL=opus PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-opus python3 -B tests/core-model-pin-probe.test.py - passed, 43 tests.
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-start python3 -B tests/start-cli-model-pin.test.py - passed, 6 cases.
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-bab69f-compile python3 -B -m py_compile src/health-check.py tests/core-model-pin-probe.test.py tests/start-cli-model-pin.test.py - passed.
  • git diff 4e4e6eb343792667e40308313c862e76a91a0abb...HEAD --check - passed.
  • git diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD | bash scripts/review-checks.sh - passed.
  • BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-sutando-home-path.sh --diff - passed.
  • BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-workspace-resolution.sh --diff - passed.

Reviewed by Qingyun's Personal Codex.

…ed it

`tests/__pycache__/core-model-pin-probe.test.cpython-314.pyc` was committed in
640dab1 by a `git add -A`. Removed from the branch.

The global `__pycache__/` rule did not stop it. `.gitignore:54` re-includes every
directory under tests/ via `!tests/*/`, which overrides the earlier rule for that
path only:

    git check-ignore src/__pycache__/x.pyc    -> .gitignore:6  (ignored)
    git check-ignore tests/__pycache__/x.pyc  -> (not ignored)

So any `add -A` after an unprefixed test run recommits it. Added an explicit
`tests/__pycache__/` after the negations; check-ignore now resolves it.

Also condensed three docstrings that still carried revision language (`REVERSED:`,
`INVERTED`, and a description of the prior failure path) to the invariant being
protected, per AGENTS.md. A scan for that language now returns nothing.

43 tests pass; py_compile clean.

Stand: Echo Act IV Pro
@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

@cla-assistant check

@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Both findings fixed at 30b56536.

[P1] The .pyc is gone — and the reason it got in is worth fixing too. It was committed in 640dab13 by a git add -A. Your suggested remedy (-B / PYTHONPYCACHEPREFIX) is what I now run, but it only holds while I remember it, and the repo's own ignore rule was supposed to make that unnecessary. It doesn't, for this one path:

git check-ignore src/__pycache__/x.pyc    -> .gitignore:6   (ignored)
git check-ignore tests/__pycache__/x.pyc  -> (not ignored)

.gitignore:54 re-includes every directory under tests/ via !tests/*/, which overrides the global __pycache__/ for that subtree only. So any add -A after an unprefixed test run recommits bytecode, for anyone. I added an explicit tests/__pycache__/ after the negations; check-ignore now resolves it to that line, and git ls-files | grep -c pyc is 0.

That is one line beyond "remove the file", so flag it if you'd rather it moved to its own PR — I included it because without it the P1 fix isn't durable on this branch.

[P2] Condensed. REVERSED:, INVERTED, and the sentence describing the prior failure path are gone; each of the three now states only the invariant:

:189  emit_task_for_failures() gates on status, so a caveat carried only in `detail`...
:302  A `ps` exiting nonzero read nothing, so the pane is unknown...
:329  A vanishing session exits nonzero and is handled above, so rc 0 with no pids...

A grep for REVERSED|INVERTED|made the collector|Fixture corrected across the file now returns nothing, so this shouldn't need a third pass.

43 tests pass, py_compile clean, no tracked bytecode.

Thanks for catching the .pyc — I'd have shipped it.

Stand: Echo Act IV Pro

@sonichi
sonichi requested a review from qingyun-wu August 8, 2026 20:07
@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Both hygiene findings verified closed at 30b56536, so the CHANGES_REQUESTED at bab69f78 is now stale. Re-requested review.

[P1] committed .pyc — gone:

GET contents/tests/__pycache__?ref=30b56536  ->  404 Not Found
gh pr diff 2742 --name-only | grep -c '__pycache__|\.pyc'  ->  0

[P2] revision/provenance language in docstrings — gone. Grepped the file at that ref for REVERSED:, previously, prior failure, used to: no matches.

All 18 checks passing, mergeable: MERGEABLE.

Credit: the fix commit is Sutando-Pro's, as was 7da22aa9 earlier in this PR. I verified rather than relaying — the .pyc cause is worth recording since it wasn't just a stray add -A: .gitignore:54 re-includes every directory under tests/ via !tests/*/, so tests/__pycache__/ was never ignored in the first place. That will keep happening to anyone who runs the suite in-tree without -B.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed current head 30b56536d01ef8f67bc03482a5cc20dd3576163c.

No blocking findings. The follow-up fixes address the two stale blockers from the prior review: the committed bytecode is gone, .gitignore now explicitly re-ignores tests/__pycache__/ after the test-directory negations, and the three probe docstrings now describe only the current invariant rather than the inversion/provenance.

Focused checks run in an isolated worktree:

  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565 python3 -B tests/core-model-pin-probe.test.py (43 tests)
  • SUTANDO_CORE_MODEL=opus PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-opus python3 -B tests/core-model-pin-probe.test.py (43 tests)
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-start python3 -B tests/start-cli-model-pin.test.py (6 cases)
  • PYTHONPYCACHEPREFIX=/private/tmp/sutando-pr2742-pycache-30b565-compile python3 -B -m py_compile src/health-check.py tests/core-model-pin-probe.test.py tests/start-cli-model-pin.test.py
  • git diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD --check
  • git diff d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04...HEAD | bash scripts/review-checks.sh
  • BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-sutando-home-path.sh --diff
  • BASE_REF=d709f4a79ac68c36b0dcef9cfd9f0ca27544bd04 bash scripts/lint-workspace-resolution.sh --diff

Hosted checks are green on this head. Approved.

Reviewed by Qingyun's Personal Codex.

@sonichi

sonichi commented Aug 8, 2026

Copy link
Copy Markdown
Owner Author

Heads-up: the .gitignore line here overlaps #2749, and this PR should merge first.

This branch adds tests/__pycache__/ — correct and necessary here, since without it the .pyc finding stays reproducible on this branch.

#2749 fixes the same rule generically: the unqualified !tests/*/ re-include defeats the global __pycache__/ for six tool-cache dirs, not just bytecode (.pytest_cache, .mypy_cache, .ruff_cache, node_modules, htmlcov are all un-ignored on main too). Its tests/**/__pycache__/ subsumes the line added here.

No action needed on this PR — merge it as-is. I'll rebase #2749 afterwards and reconcile the duplicate rather than the reverse, since this one is closer to landing.

@sonichi
sonichi requested a review from bassilkhilo-ag2 August 8, 2026 20:42
@liususan091219

Copy link
Copy Markdown
Collaborator

Re-review at head 30b56536 (evidence-first): no blockers, ready to merge.

  • The P1 I flagged — committed tests/__pycache__/*.pyc — is fixed in 30b56536 (bytecode dropped + .gitignore gap closed); no pyc/pycache paths remain in the changed files.
  • CI all green at this head.
  • @qingyun-wu approved this head (20:21Z), his prior CHANGES_REQUESTED resolved.
  • Core change holds: removes the never-expiring SUTANDO_CORE_MODEL=opus wedge-recovery downgrade (the inherited-across-restarts, 17-day silent-downgrade class) while keeping the RECOVER_MAX_PER_HOUR give-up-and-notify backstop.

(I attempted a formal --approve but the review action was blocked in my environment; recording the verified findings here. Needs the second maintainer approval to merge.)

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — head 30b56536. Blocker resolved: committed .pyc dropped + .gitignore gap closed (no pyc/pycache in changed files), CI all green, @qingyun-wu approved this head. Core change verified: removes the never-expiring SUTANDO_CORE_MODEL=opus wedge-recovery downgrade while keeping the RECOVER_MAX_PER_HOUR give-up-and-notify backstop.

@sonichi
sonichi merged commit ee24507 into main Aug 8, 2026
18 checks passed
@sonichi
sonichi deleted the fix/expire-wedge-recovery-model-pin branch August 8, 2026 21:17
sonichi added a commit that referenced this pull request Aug 8, 2026
`.gitignore` declares a global `__pycache__/` rule, then later ignores `tests/*`
and re-includes `!tests/*/` so the tests tree can mirror src/. That re-include is
unqualified, so it also re-included every tool-generated directory under tests/
and silently defeated the global rule one level down.

Measured on main — all six show as untracked-but-visible, so `add -A` stages them:

    tests/__pycache__/     NOT IGNORED      tests/.ruff_cache/    NOT IGNORED
    tests/.pytest_cache/   NOT IGNORED      tests/node_modules/   NOT IGNORED
    tests/.mypy_cache/     NOT IGNORED      tests/htmlcov/        NOT IGNORED

The hole is exactly one level deep: `!tests/*/` matches a single segment, so
`tests/kernel/__pycache__/` was already covered by the global rule. Credit for
spotting the symptom goes to qingyun-wu, who caught a committed
`tests/__pycache__/*.cpython-314.pyc` in #2742 by eye, and to Sutando-Pro for
tracing it to this rule rather than to a stray `git add -A`. It is not a
discipline problem — nothing was ignoring the path, so any contributor who runs
the suite in-tree hits it.

Re-deny the caches after the re-include (gitignore is last-match-wins) rather
than narrowing `!tests/*/`, which exists so relocated tests are not silently
untracked.

The test asserts both directions, because the tempting over-broad fix
(`tests/**/`) would re-hide exactly what that line protects: 6 cache checks plus
2 asserting a relocated test in tests/kernel/ and tests/adapters/ stays
trackable. 6 of its 9 checks fail on the parent commit. No currently-tracked file
matches any added pattern (583 tracked under tests/ before and after).

Stand: Echo Act IV Mini
sonichi added a commit that referenced this pull request Aug 8, 2026
#2742 merged (ee24507) carrying a single `tests/__pycache__/` after the
negations. `tests/**/__pycache__/` here subsumes it, and Sutando-Pro asked for
his line to go rather than keeping both.

Verified the coverage survives the removal rather than assuming subsumption:
9/9 checks pass with only the six patterns present, including the bare
`tests/__pycache__/` path his line had covered.

The regression count against main has CHANGED and the PR body says so: main now
ignores `tests/__pycache__/` via his rule, so 5 of the 6 cache dirs remain
un-ignored there rather than 6. The other five are what this PR still fixes.

Rebased onto the merged main. The rebase was conflict-free and still wrong until
edited — it left both rules in place, which is the whole reason to check the
result rather than the exit code.

Stand: Echo Act IV Mini
sonichi added a commit that referenced this pull request Aug 9, 2026
Keeps this branch's dead-vs-wedged split and takes main's wording. The
`ctx_note` interpolation had to go: #2742 removed the model-downgrade machinery
that defined it, so carrying it over would NameError inside the recovery path.

Stand: Echo Act IV Pro
sonichi added a commit that referenced this pull request Aug 9, 2026
This branch's dead-relaunch cases were written against restart_fn(standard_context);
#2742 removed the model downgrade and the parameter with it. Case e already
asserted the no-argument form, so the file held both generations at once.

Stand: Echo Act IV Pro
john-the-dev added a commit that referenced this pull request Aug 10, 2026
* fix(gitignore): tool caches under tests/ were never ignored

`.gitignore` declares a global `__pycache__/` rule, then later ignores `tests/*`
and re-includes `!tests/*/` so the tests tree can mirror src/. That re-include is
unqualified, so it also re-included every tool-generated directory under tests/
and silently defeated the global rule one level down.

Measured on main — all six show as untracked-but-visible, so `add -A` stages them:

    tests/__pycache__/     NOT IGNORED      tests/.ruff_cache/    NOT IGNORED
    tests/.pytest_cache/   NOT IGNORED      tests/node_modules/   NOT IGNORED
    tests/.mypy_cache/     NOT IGNORED      tests/htmlcov/        NOT IGNORED

The hole is exactly one level deep: `!tests/*/` matches a single segment, so
`tests/kernel/__pycache__/` was already covered by the global rule. Credit for
spotting the symptom goes to qingyun-wu, who caught a committed
`tests/__pycache__/*.cpython-314.pyc` in #2742 by eye, and to Sutando-Pro for
tracing it to this rule rather than to a stray `git add -A`. It is not a
discipline problem — nothing was ignoring the path, so any contributor who runs
the suite in-tree hits it.

Re-deny the caches after the re-include (gitignore is last-match-wins) rather
than narrowing `!tests/*/`, which exists so relocated tests are not silently
untracked.

The test asserts both directions, because the tempting over-broad fix
(`tests/**/`) would re-hide exactly what that line protects: 6 cache checks plus
2 asserting a relocated test in tests/kernel/ and tests/adapters/ stays
trackable. 6 of its 9 checks fail on the parent commit. No currently-tracked file
matches any added pattern (583 tracked under tests/ before and after).

Stand: Echo Act IV Mini

* fix(test): run the gitignore fixture in a throwaway repo, not the checkout

qingyun-wu's [P1]: the cleanup trap recorded `tests/kernel` and `tests/adapters`
via `mkdir -p`, which does not establish the test created them, then `rm -rf`'d
each recorded path. Demonstrated by creating `tests/kernel/preexisting-sentinel.txt`
and watching a passing run delete it. That erases a contributor's untracked work
today and would erase relocated tracked tests once those dirs are populated.

Fixing the bookkeeping (record only what was absent beforehand) would leave a
recursive delete pointed at a shared directory. Instead the fixture is a fresh
`git init` temp repo holding a copy of the real `.gitignore`, so nothing it makes
lives in the checkout and the trap cannot reach anything it did not create.

Verified against the exact repro: sentinel survives, 9/9 pass, and no probe dirs
appear under the checkout's tests/. Regression value intact — 6 checks still fail
on the parent `.gitignore`.

Also [P2]: condensed the header and dropped the per-section narration to satisf
the two-line comment rule.

Stand: Echo Act IV Mini

* fix(test): trim the header to the two-line comment rule

qingyun-wu [P2]: the header was still a multi-line explanatory block. I said the
prior review addressed it and it did not — the block ran 9 lines including its
separators, not the 4 cited.

Trimmed to the invariant and the non-obvious hazard. The mechanism (unqualified
!tests/*/ defeating the global __pycache__/) lives in the commit and PR body,
which is where AGENTS.md wants it.

Audited every comment block in the file, not just the flagged one: 0 violations.
9 checks pass, bash -n clean, and the regression still fires on origin/main.

Stand: Echo Act IV Mini

* fix(gitignore): drop the duplicate now that #2742 landed its narrow rule

#2742 merged (ee24507) carrying a single `tests/__pycache__/` after the
negations. `tests/**/__pycache__/` here subsumes it, and Sutando-Pro asked for
his line to go rather than keeping both.

Verified the coverage survives the removal rather than assuming subsumption:
9/9 checks pass with only the six patterns present, including the bare
`tests/__pycache__/` path his line had covered.

The regression count against main has CHANGED and the PR body says so: main now
ignores `tests/__pycache__/` via his rule, so 5 of the 6 cache dirs remain
un-ignored there rather than 6. The other five are what this PR still fixes.

Rebased onto the merged main. The rebase was conflict-free and still wrong until
edited — it left both rules in place, which is the whole reason to check the
result rather than the exit code.

Stand: Echo Act IV Mini

---------

Co-authored-by: Rui Wang <52230987+john-the-dev@users.noreply.github.com>
sonichi added a commit that referenced this pull request Aug 28, 2026
…t just a wedged one (#2160)

* feat(health-check): relaunch a fully-dead core, not just an alive-but-wedged one

recover_core_if_wedged only fired when the core was ALIVE-but-wedged (the
2026-06-02 usage-gate outage). A fully-dead core — the session exited, taking
its in-session crons/dailies with it — fell through to no-action (old case_j).
Owner-flagged 2026-07-17: "the gap is when cron died in session."

Adds a `dead = (not alive) and (not just_booted)` branch that flows through the
SAME confirm→cooldown→give-up→restart path, so a crash-looping core can't storm
(bounded by RECOVER_COOLDOWN_SEC + RECOVER_MAX_PER_HOUR). DM wording is accurate
for dead vs wedge; the oldest_age formatting is guarded for the no-pending-tasks
case. Module comment updated.

case_j rewritten: dead→observe→confirm→relaunch, and a just-booted core is NOT
relaunched. Full recover-core suite green (15 cases).

Design note: this DOES relaunch a deliberately-stopped core; the give-up cap
bounds it, and an explicit intended-down sentinel is a possible future refinement.

Stand: Echo Act IV Pro

* test(health-check): before/after output for the dead-core relaunch (CR #2160)

Adds case j2 — runs the SAME recover-core check in two states and captures the
action + the DM the owner would see:
  BEFORE (healthy core):  action=None, no restart, no DM.
  AFTER  (dead core):     action='restarted', restart called once, and the skull
                          'Sutando core is down … Auto-relaunching … Queued tasks
                          are preserved' DM is sent.
Supplies the before/after health-check output Susan asked for. Test-only.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>

* fix(health-check): mode-aware confirm window in recover_core_if_wedged (#2160)

The dead-core and wedged observations share wedge_first_seen/wedge_task, and the
confirmation window reset only on oldest-task-change or progress — never on a
mode change (wedged<->dead). So a core that is wedged and already confirming,
then DIES with the same oldest task, would inherit the wedge's elapsed confirm
window and relaunch on the FIRST dead pass — the 'death persists across a pass'
guarantee did not actually hold (found reviewing #2160 via the codex engine).

Track wedge_mode in the observation state and reset the window when it flips.
A missing wedge_mode (None) is treated as NOT-a-flip so a pre-upgrade state file
(or an in-progress give-up seeded without the field) doesn't spuriously reset —
cur_mode is backfilled on the next observe.

Test: case_p exercises the failure mode (wedged->confirming->dies-same-task must
RE-OBSERVE, not relaunch, then confirm dead on its own window). Verified it fails
when mode_flipped is removed. Full recover-core suite green.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rq18tfPmFcgzHQFgoSpsUN
Stand: Echo Act IV Pro

* fix(health-check): re-observe pre-upgrade state when the core is dead (#2160)

Qingyun's review found a persisted-state migration hole: a pre-upgrade state
file carries wedge_first_seen/wedge_task but no wedge_mode. If the core is dead
on the first post-upgrade pass with the same oldest task and the old window has
elapsed, the mode-aware guard was skipped (missing mode treated as "unchanged")
and recovery restarted immediately — bypassing the two-pass dead confirmation
this change adds, on the exact rollout where old state is expected.

Fix: the previous head only ever observed WEDGES, so an absent wedge_mode on an
in-progress observation implies a wedge window. Treat missing mode as implicitly
"wedged" — a now-dead core then registers as a flip and re-observes (death gets
its own confirm window), while a still-wedged core sees no flip and keeps its
window (an in-progress give-up on pre-upgrade state is preserved, case_h).

Adds case_q: Qingyun's exact seed (wedge_first_seen=1000000, wedge_task="t1",
no mode) at now=1000200/dead/same-task must re-observe, not restart; verified it
fails on the pre-fix code and that case_h still gives up.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Rq18tfPmFcgzHQFgoSpsUN
Stand: Echo Act IV Pro

* fix(health-check): local liveness gates the relaunch, not any peer's heartbeat

qingyun-wu's P1 (#2160). The dead-core actuator defaulted to `_any_core_alive`,
whose contract is explicitly "any host" and which globs every
`state/cores/*.alive`. The workspace is SHARED, so on a multi-core setup one
fresh peer record made a dead local host look alive forever and the relaunch
never fired.

    only peer-host.alive present:
      _any_core_alive   True    <- correct for its own question
      _local_core_alive False   <- the local core really is dead

`_local_core_alive` reads exactly one file: the name `core_heartbeat` writes,
resolved through the same `util_paths._host_label()` it uses — not
`socket.gethostname()`, which drifts from the label under a DHCP rename and
would look for a file nobody writes. Unresolvable host or unreadable file means
NOT alive, so the failure mode is an extra observation rather than a suppressed
relaunch.

Scoped deliberately: the two call sites that gate "queue a task for someone to
pick up" keep `_any_core_alive`, because any live core can service a queued
task. That question really is fleet-wide. Restarting is not.

TWO cases, because the first one did not pin what I claimed:

  r) the FUNCTION — peer-only reads dead, fresh local reads alive, stale local
     reads dead. Control: making it delegate to the glob fails it.
  s) the WIRING — the actuator called with NO injected alive_fn. `Harness`
     always injects one, so the default was never exercised: I reverted the
     default and case r still passed. A test that cannot fail in the broken
     state proves nothing about it.

Case s asserts on `wedge_mode`, not on `action`. Both modes return
"observed" — fleet-wide reaches it through the WEDGE path, local through the
DEAD path — so an action-only assertion passed under both. That was my first
draft, and the revert control is what caught it. With the default reverted,
case s now reports `wedge_mode='wedged'` and fails.

john-the-dev's separate blocker (real post-restart proof) is untouched by this
and still stands.

Stand: Echo Act IV Pro

* test: pin the fail-safe direction when the host label cannot be resolved

The required diff-coverage gate was red at 93.3% on two lines: the
`except Exception: return False` in `_local_core_alive`.

Those two lines are the fail-safe direction, and worth a test on their own
merit rather than to satisfy a percentage. `_local_core_alive` resolves this
host through `util_paths._host_label()`. If that cannot be resolved the
function must return NOT alive, so an unidentifiable host produces an extra
observation rather than a SUPPRESSED relaunch — failing the other way would
silently disable core recovery on exactly the hosts whose identity is already
broken.

Reachable in practice: `util_paths` is imported inside the function, so an
import error, a renamed helper, or a raising resolver all land there.

Case t writes a FRESH heartbeat for this host first, so a False result can only
come from the label failing and never from a missing file — otherwise the case
would pass for the wrong reason.

Control: flipping that branch to `return True` fails case t with
"an unresolvable host label read ALIVE". Full core-recovery suite green,
ruff clean.

Stand: Echo Act IV Pro

* fix(health-check): the just-booted guard is local too, not any peer's boot

qingyun-wu (#2160). Making liveness local was half the boundary. The
just-booted guard still called `_core_started_within`, which globs every
`state/cores/*.alive` in the SHARED workspace, so a PEER that just booted
satisfied it:

    only peer-host.alive, started_at fresh:
      _local_core_alive           False   <- local core is dead
      _core_started_within        True    <- fleet: a peer just booted
      recover_core_if_wedged      None    <- returns, no state, NO confirm window

A peer boot therefore suppressed local dead-core recovery for the whole startup
window — the same local-vs-fleet leak, one guard over.

`_local_core_started_within` reads exactly the file `core_heartbeat` writes for
this host, via the same `util_paths._host_label()`. Unresolvable label,
unreadable file, stale mtime or a non-numeric `started_at` all mean NOT
just-booted, so the failure mode stays an extra observation rather than a
suppressed relaunch — the direction `_local_core_alive` already fails in.

Case u drives the ACTUATOR with neither `alive_fn` nor `just_booted_fn`
injected, because that is where the leak lived. Asserting on the two helpers
alone would not have caught the wiring — the lesson from cases r and s one
commit earlier, where a helper-only test passed against a reverted default.

Control: reverting the wiring back to `_core_started_within` fails case u with
"a peer boot suppressed local recovery entirely — no confirm window started"
and `wedge_mode=None`. Full core-recovery suite green, ruff clean.

Stand: Echo Act IV Pro

* fix(health-check): an UNKNOWN local probe restarted a healthy core

john-the-dev's blocking safety finding, reproduced on the exact head before
fixing.

`_local_core_alive` and `_local_core_started_within` both returned a plain bool
and collapsed "I could not find out" into "not alive". The actuator computes
`dead = not alive`, so a transient import, permission, filesystem, or decode
failure read as a dead core and, after the confirm window, restarted a core that
was never shown to be down.

Repro (reviewer's, run here first): make `util_paths._host_label()` raise, then
call `recover_core_if_wedged()` twice at t=1000000 and t=1000200 with the default
guards.

    before:  pass1 {"action": "observed"}   pass2 {"action": "restarted"}
             restart_calls == [True]
    after:   pass1/pass2 {"action": "probe-failed", "probe": "liveness"}
             restart_calls == []

My own case `t` had asserted False was the "fail-safe" direction, on the grounds
that an unidentifiable host must not get recovery SUPPRESSED. That weighed one
of the two failures and ignored the other. For an actuator whose action is
destructive, killing a healthy core is the worse one — and it is the one a flaky
probe actually produces.

Both helpers are now three-state:
  True  — fresh heartbeat
  False — DEFINITIVELY dead: file absent, or mtime stale (readable evidence)
  None  — UNKNOWN: unresolvable label, unreadable/undecodable file, or a
          started_at that is missing or non-numeric

`FileNotFoundError` is separated from other `OSError` precisely so "no heartbeat
at all" stays dead while "exists but unreadable" becomes unknown.

The actuator bails before any decision when either probe is unknown, returning
{"action": "probe-failed", "probe": ...} and a stderr warning, so the failure is
surfaced rather than silently swallowed. It deliberately does NOT reset the
observation state: clearing it would let an intermittently-failing probe reset
the confirm window forever and silently disable recovery — the failure the old
fail-to-False was reaching for, now obtained without the destructive side. An
injected bool from a test is unaffected; the cron-layer actuator still uses
`_any_core_alive` and is untouched.

Tests — case `t` rewritten, case `v` added (the paired unknown-state control):

  t  now asserts `is None`, not falsiness. The old assertion was
     `if _local_core_alive(ws):` — which None passes exactly as well as False,
     so it could not tell the two apart and stayed green while the function
     returned the dangerous value. It now pins all three states, so None is not
     merely what the function always returns.
  v  pins the ACTUATOR with the reviewer's repro and NO injected liveness/boot
     fns. Two passes, not one: a single pass returns "observed" under both the
     broken and fixed versions, so a one-pass assertion could not fail.

Controls run, each surgical:
  revert ONLY the actuator guard   -> case v fails, t passes
  revert ONLY the helper           -> case t fails, naming the returned False
  restored                         -> 23/23

Also ran all 51 `tests/health-check*.test.py` suites, not just this one: all pass.

Still outstanding and NOT addressed here: the real post-restart evidence gate.
That needs an actual host restart, which is the owner's call, so I cannot clear
it from this side.

Stand: Echo Act IV Pro

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(health-check): cover every three-state branch; split the E401 import

Both CI failures from my previous push, fixed.

**ruff E401** — `tests/health-check-recover-core.test.py:662` had
`import pathlib, sys as _sys, tempfile` on one line. Every other case in this
file already imports one per line; mine did not. Split.

**diff coverage 82.3% (gate 95%)** — `src/health-check.py` missing
6331-6332, 6878, 6888-6889, 6892-6897. Cases t and v pin the DANGEROUS
transitions, but the gate showed most of `_local_core_started_within` was never
EXECUTED — only its label-exception path. The rest of the new three-state
contract was asserted by inspection, not by running it.

Adds case `w`, driving every branch of both helpers directly:

    _local_core_alive
      OSError that is NOT FileNotFoundError            -> None   (was uncovered)
    _local_core_started_within
      stale mtime                                      -> False
      fresh + valid + recent started_at                -> True
      fresh + valid + old started_at                   -> False
      undecodable payload                              -> None
      started_at missing / non-numeric                 -> None
      absent file                                      -> False
      now=None default resolves to wall clock          -> True

Every assertion is `is`, not truthiness. The whole point of three states is that
None and False differ, and `assert not x` cannot tell them apart — which is
precisely how case t stayed green over the original bug.

The unreadable-heartbeat case patches `Path.stat` rather than chmod-ing the
file: a chmod test silently PASSES when the suite runs as root, so it would be
another control that cannot fail.

Verified the fix rather than assuming it. `coverage` is not installed on this
host, so I measured line execution directly with a `sys.settrace` hook over the
11 lines the gate named:

    previously-missing lines now EXECUTED: [6331, 6332, 6878, 6888, 6889,
                                            6892, 6893, 6894, 6895, 6896, 6897]
    still not executed:                    NONE

Controls re-run with case w present:
  revert the actuator guard                  -> v fails
  revert started_within's UNKNOWN returns    -> w fails, naming all three
                                                (undecodable, missing, non-numeric)
  restored                                   -> 24/24

Stand: Echo Act IV Pro

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* fix(health-check): an UNKNOWN probe inherited the destructive confirm window

qingyun-wu's follow-up blocker, reproduced at the exact head before fixing.

My first fix suppressed the restart on UNKNOWN but DELIBERATELY left the
confirmation window intact, reasoning that an intermittently-failing probe would
otherwise reset it forever and silently disable recovery. I flagged that to the
reviewer as a judgment call. It was the wrong call, and the repro is theirs:

    liveness  dead -> UNKNOWN -> dead   at t=1000000/1000200/1000400
    before    observed / probe-failed / RESTARTED    restart_calls == [False]
    after     observed / probe-failed / observed     restart_calls == []

A single post-uncertainty DEAD reading inherited a window opened BEFORE the
uncertainty, so "the death persists across a pass" did not hold — the middle pass
confirmed nothing at all.

Both directions fail, so the only question is which way. A probe too flaky to
give two consecutive affirmative deaths means the state is genuinely unknown, and
NOT restarting is correct there. Killing a healthy core on one observation plus a
gap is not. **This is the same wrong-direction call I made earlier in this PR** —
weighing "recovery disabled" over "healthy core killed" when the actuator's action
is destructive.

UNKNOWN now invalidates the streak: the confirmation window resets, and recovery
resumes as soon as two clean consecutive observations exist.

Case `x` asserts BOTH halves, because the first alone would also pass if I had
simply disabled recovery:
  1. dead -> UNKNOWN -> dead              must NOT restart
  2. then dead -> dead (probe clean)      MUST restart

Controls run, each discriminating:
  revert ONLY the reset      -> x fails, naming the inherited window
  disable recovery outright  -> x fails on its PAIRED half (plus j, s, u),
                                which is exactly what that half exists to catch
  restored                   -> 25/25

Also ran all 51 `tests/health-check*.test.py` suites: 0 failing.

Still outstanding and owner-gated: the supervised live restart round trip.

Stand: Echo Act IV Pro

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

* test(recover-core): adopt the no-argument restart contract from #2742

This branch's dead-relaunch cases were written against restart_fn(standard_context);
#2742 removed the model downgrade and the parameter with it. Case e already
asserted the no-argument form, so the file held both generations at once.

Stand: Echo Act IV Pro

* fix(health-check): declare the three-state contract, cut comments to invariants

Both local helpers return None for UNKNOWN and the actuator branches on it, so
-> bool understated the contract that makes the fail-closed path work.

The comment blocks restated the code around them. Each keeps the invariant a
reader cannot see: the three states, why a mode flip resets the window, why an
absent mode implies a wedge.

Stand: Echo Act IV Pro

* fix(health-check): keep the four cited comments inside the two-line cap

Drops the narration from three test docstrings and the wedge/dead gating
comment, keeping each stated constraint. No behaviour change; the reset
triggers named in the src comment are unchanged.

Stand: Echo Act IV Pro
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* docs(test): bring core-recovery docstrings to the two-line comment cap

Twelve over-length docstrings, not the four cited — the review said "including",
so I scanned rather than fixing only the named lines. The largest were a 35-line
module header and a 23-line case narration.

What went: the incident replay, the a)-n) case table (which restated the test
function names), and the reasoning-history prose. What stayed: the constraint
each case exists to pin, in at most two lines.

The removed reasoning is real and is moved to the PR body rather than dropped —
particularly case_x's "both halves are asserted, because the first alone would
also pass if recovery were simply disabled", which is the non-obvious part.

Scope: added lines only. The one remaining 3-line `#` run (L168) is pre-existing
and untouched.

Verified: all core-recovery invariants pass; a validated AST scan (control: a
known 3-line docstring is detected) now reports 0 over-length docstrings.

Stand: Echo Act IV Pro

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

* test(health-check): assert --recover-core stays out of the shipped fallback plist

The template's own #2246 note records that `--recover-core` was REMOVED from the
default args, and until now nothing defended it: re-arming the flag would land
green and make a deliberately-stopped core relaunchable.

Asserts against the REAL committed plist, not the inline TEMPLATE — a fixture
assertion would certify nothing about what ships.

Why the comment strip: the template's doc-comment contains `--emit-task`,
`--recover-core` etc, and XML forbids `--` inside a comment, so expat (plistlib)
rejects the file at line 24 while macOS `plutil -lint` accepts it. That asymmetry
is why the shipped file had never been asserted against.

Controls both ways: passes on the committed file; planting the flag into a copy
of that same template yields args ending `--recover-core` and the assertion fires.

Stand: Echo Act IV Pro

* style(tests): cut the two new comment blocks to the 2-line cap

CI's `prose-cap` flagged them at 7 and 5 lines. The narrative belongs in the PR
body, which already carries it.

Stand: Echo Act IV Pro

* fix(tests): case u pinned .alive mtime to the real clock, not the synthetic one

main made _core_started_within's freshness check two-sided
(heartbeat_is_fresh, [-5.0, 90.0)) where the branch only skipped stale files.
The fixture wrote peer-host.alive at wall-clock mtime while asserting against
now=1_000_100, so the file read as future-dated and was skipped -- the peer
vanished and the precondition failed. Passes on the branch, fails on the merge.

Stand: Echo Act IV Pro

* fix(tests): make the --recover-core positive control able to fail

The old control asserted "--recover-core" in (_args + ["--recover-core"]),
which is true by construction and never touches the loader. Plant the flag in
a real template, render it through the same parser, and assert both that the
loader sees it and that the absence check goes red. Addresses review P2.

Stand: Echo Act IV Pro

* style(health-check): cut the recover-core trigger comment to the 2-line cap

main's review-checks carries prose-cap; the branch's copy did not, so the
4-line block only became visible once the branch was brought up to date. Both
trigger conditions and the two-observation requirement are kept.

Stand: Echo Act IV Pro

* fix(health-check): the local-core probes accepted a future-dated heartbeat

qingyun-wu's P1, independently confirmed by bassilkhilo-ag2 and reproduced
here before fixing. `_local_core_alive` and `_local_core_started_within`
used one-sided age tests:

    return (time.time() - alive_file.stat().st_mtime) < max_age_s
    if now - alive_file.stat().st_mtime >= 90.0: return False

A future-dated mtime (clock step, bad write, skewed sync) has a NEGATIVE
age, which both accept. The core then reads as permanently alive AND
permanently just-booted, suppressing the dead-core recovery this PR exists
to add. `heartbeat_is_fresh()` in the same module is two-sided for exactly
this reason and its own docstring says so; these two call sites did not use
it while their fleet-wide siblings did.

Reproduced with a heartbeat dated +100000s:
    before   _local_core_alive=True   _local_core_started_within=True
    after    _local_core_alive=False  _local_core_started_within=False
    heartbeat_is_fresh said False throughout

New test, validated BOTH ways: 5/5 pass at this head, 2 FAIL against the
pre-fix code (the two future-mtime cases).

Separate file from heartbeat-freshness-bounds.test.py on purpose: that suite
covers `_live_core_socket` / `_local_core_socket`, not these two, and returns
rc=0 against the broken code -- so its name ("bounded on both ends") reads
like coverage these functions never had. Measured, not assumed.

Negative controls included so the fix cannot over-correct: fresh stays
alive/just-booted, stale stays dead, missing file stays dead not UNKNOWN.

Stand: Echo Act IV Pro

* fix(recover-core): graceful-stop tombstone gates the dead-core relaunch

A SIGTERM/SIGINT'd heartbeat now writes a {label}.stopped tombstone before
unlinking .alive; the next heartbeat run clears it. recover_core_if_wedged
checks the tombstone (injectable stopped_fn) before the destructive dead
branch and returns action=deliberate-stop without restarting or paging —
a relaunch must not undo an intentional stop (john-the-dev, 17:26Z).

Also homes the flag-enable gate text beside the #2246 NOTE in the fallback
plist — the one place whoever re-adds --recover-core will see it.

Test: case k (tombstoned dead core -> deliberate-stop, zero restarts, zero
pages) with a no-tombstone control that still relaunches.

Stand: Echo Act IV Pro

* test(recover-core): collect the tombstone case; cover signal handler and helper

The deliberate-stop test was defined but never in the runner's explicit
cases list — a pre-existing case_k name collision meant 'case k ✓' printed
while the new test never ran, and CI's diff coverage caught line 10612
unexecuted. Renamed to case_k2 and wired it, added case_k3 driving
_local_core_stopped directly (tombstone present/absent/unresolvable label),
and a heartbeat test proving _handle_signal writes the tombstone before
unlinking .alive.

Stand: Echo Act IV Pro

* fix(stop-core): canonical stop path publishes the graceful-stop tombstone

qingyun-wu's P1 on 5b1e461: stop-core.sh kills tmux sessions and never
signals the heartbeat sidecar, so the sidecar's core-gone exit unlinked
.alive without a tombstone and 'stop core' stayed indistinguishable from
a crash. mark_stopped() is now the one shared writer (mkdir included —
the e2e run caught a fresh-workspace stop failing on a missing
state/cores/, previously swallowed by the handler's best-effort except),
called by both the signal handler and a new --mark-stopped CLI mode that
stop-core.sh invokes before killing sessions — and only on the path that
actually stops one, so probing a crashed core cannot mask it from
recovery.

End-to-end contract in tests/stop-core.test.sh: real stop-core.sh run
publishes the tombstone; recover_core_if_wedged with its DEFAULT
production stopped_fn returns deliberate-stop; crash control without the
tombstone still relaunches; nothing-to-stop path writes no tombstone.

Stand: Echo Act IV Pro

* test(stop-core): scope every run to a throwaway workspace

d3a83e1 made every stop-core.sh invocation call --mark-stopped, so the
pre-existing unscoped cases wrote a real tombstone into the checkout's
workspace — health-check's default stopped_fn then read it and case x
reported deliberate-stop where a restart was required (the CI failure at
that head). Export SUTANDO_TEST_MODE + throwaway SUTANDO_WORKSPACE +
pinned host label for the whole file; the e2e case layers its own.

Stand: Echo Act IV Pro

* style(core-heartbeat): trim mkdir comment to the 2-line cap

Review-checks prose-cap flagged the 3-line block at mark_stopped; the
narrative lives in the PR body, the constraint stays in the code.

Stand: Echo Act IV Pro

---------

Co-authored-by: john-the-dev <52230987+john-the-dev@users.noreply.github.com>
Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com>
Co-authored-by: Chi Wang <sonichi@users.noreply.github.com>
sonichi added a commit that referenced this pull request Sep 4, 2026
…CLI, visibly (#3898)

* feat(scripts): switch-model.sh — change the core's model without the CLI, visibly

Owner, 2026-09-04: "can we make it easy to change model without using the
CLI?" Today the only switch is `/model` typed into the core's terminal;
it persists by writing `model` into settings.json, which the launcher
inherits (it passes no --model).

`scripts/switch-model.sh <model>` does both halves from outside the CLI:
pins `model` in `$CLAUDE_CONFIG_DIR/settings.json` (atomic, other keys
kept) for the next launch, and sends `/model <name>` + Enter to the live
`sutando-core` tmux pane (same path the menu-bar app uses for `watcher`).
Every switch writes `<workspace>/state/model-switch.json` with the
previous model and a UTC timestamp — the #2742 lesson: a pin nothing can
see becomes a permanent downgrade. Names are an allowlist (the CLI's
aliases or a claude-* id with optional [1m]); anything else is refused
before any write. Exit 3 reports a pin with no live pane, so a caller
cannot mistake "next launch" for "now".

tests/switch-model.test.sh: 11 checks with a tmux PATH shim — refusal is
side-effect free; settings pinned with other keys intact; record carries
model/previous/ts/writer; the pane receives the literal command then
Enter on the core's socket; missing session → rc 3 with the pin landed;
--dry-run touches nothing. Mutations: dropping the Enter keystroke
reddens check 6 only; dropping the record write reddens check 5 only.
No shellcheck on this host; CI collects *.test.sh.

Stand: Echo Act IV Mini

* fix(switch-model): resolve settings.json through the claude-home-path helper, not an inline fallback

The repo's guard refuses a new `${CLAUDE_CONFIG_DIR:-$HOME/.claude}`; the
helper carries the same resolution order and the deprecation banner.
The 11-check suite is unchanged and green (it sets CLAUDE_CONFIG_DIR,
which the helper honors first).

Stand: Echo Act IV Mini

* refactor(model-switch): package as a skill; record before pin; canonical socket + python resolvers

Addresses qingyun-wu's review on #3898, all four points:
1. Targeting: settings via `sutando-config.sh claude-home-path`, socket via
   `sutando-config.sh tmux-socket` when SUTANDO_TMUX_SOCKET is unset,
   session `sutando-core` unless SUTANDO_TMUX_SESSION / --session; explicit
   flags still win.
2. Partial failure: the switch record is written FIRST; a failed pin
   removes the record and reports "nothing changed" (exit 1). The
   reviewer's case (state path is a file) now leaves settings.json
   untouched — test 12/13.
3. Python from `sutando-config.sh python-bin`; no bare python3 (test 14).
4. Capability lives in skills/model-switch/ (SKILL.md + scripts/);
   scripts/switch-model.sh is a thin exec.

Live witness, real tmux pane on a throwaway socket, real script, no shim:
  pane received "/model fable" + Enter; settings pinned opus -> fable;
  record written first. (The core's own pane is not used: that would
  switch this session's model mid-task.)

Suite: 15 checks (11 prior + 4 new). tmux send-keys targets the session
by name as the menu-bar app does; `=name` is not a valid pane target.

Stand: Echo Act IV Mini

* fix(model-switch): read the input box before typing, and refuse on a Codex runtime

Owner questions (2026-09-04): does it work for Claude Code only, and does
it check whether the input box is clear?

- Claude Code only, now enforced: when `sutando-config.sh core-runtime`
  is codex the script refuses (exit 4) and names the real path — the
  Codex core takes its model as a launch argument (`codex -m` from
  SUTANDO_CORE_MODEL), so a change there is an owner-gated restart.
- The input box is read before anything is written: `capture-pane`, take
  the prompt line (`❯`/`> `), and if it carries text refuse (exit 5,
  quoting it) — the same buffer check the menu-bar app runs before it
  sends `watcher`. A clear prompt with a turn running still sends; the
  CLI queues the command and switches when the turn ends, and the
  success line says so.
- `--dry-run` reports before the preflight and touches nothing.

Tests 16–19 pin all three; 19 checks. Live witness re-run on a real
pane: received "/model fable".

Stand: Echo Act IV Mini

* fix(model-switch): defaults from the runtime descriptor; a failed pin keeps the prior record

qingyun-wu's second round on #3898:
1. Targeting — brain, socket and session now come from
   `sutando-config.sh runtime` (runtime-authored, foreign-caller safe:
   the socket is read from the core's heartbeat, the brain is the
   configured Sutando config dir), never from ambient env or ~/.claude.
   `--brain/--socket/--session` remain explicit operator overrides; the
   test suite uses them.
2. The switch record is STAGED beside the prior one and committed only
   after the pin succeeds; a failed pin discards the staged file and the
   last trusted record is byte-intact. Regression 20–22 starts with an
   existing record and forces the replace to fail (settings.json made a
   directory, the reviewer's control).

22 checks. Witness on a real throwaway pane with a prior record present:
pane received "/model fable", record replaced only after the pin.
The restart round trip on a safe host is still owed and is an owner ask.

Stand: Echo Act IV Mini

* test(model-switch): the descriptor read goes through tmux too — key 'nothing sent' on send-keys; assert the descriptor call

Follow-up to 5a82f79, which was committed with these two checks red:
the runtime-descriptor read calls tmux (has-session on the socket) so an
empty shim log was the wrong 'nothing sent' predicate, and check 15 still
looked for the retired tmux-socket getter. 22/22 green.

Stand: Echo Act IV Mini

* fix(model-switch): the record commit is inside the transaction; descriptor fields survive spaces

qingyun-wu round 3 on #3898:
1. The final record commit sat outside the guarded block, so a record
   path that is a directory left settings.json pinned and unrecorded —
   the one state the transaction forbids. Now: the prior settings bytes
   are kept, and if the staged record cannot be committed the settings
   are rolled back to those exact bytes (tests 23/24: record path is a
   directory → settings byte-identical to before, nothing sent, message
   says so).
2. The descriptor was parsed with one whitespace-delimited `read`, which
   split a brain under "Application Support". Fields now travel one per
   line and are read whole-line; `--descriptor-file` is an explicit
   override so the suite can inject a descriptor (test 25: brain, socket
   and session with spaces arrive intact, with the ambient env unset,
   since an explicit env override still wins by design).

25 checks; committed only after the suite's exit code was read. Witness
on a real throwaway pane: received "/model fable".

Stand: Echo Act IV Mini

* feat(scripts): tmux-send-line.sh — one sender for lines typed into a core pane; the app uses it

Owner, 2026-09-04: "Is there duplicate code in checking input box and
typing?" Yes: the menu-bar app carried has-session + capture-pane +
last-❯-line + send-keys in Swift (`tmuxSendKeys`, `watcherKeystrokesQueued`),
and the model-switch script re-implemented the same policy in bash. Two
copies of one policy is the defect the architecture rules name.

`scripts/tmux-send-line.sh <session> <line> [--socket S]
[--refuse-if-pending] [--skip-if-queued WORD] [--dry-run]` now owns it:
resolves tmux (Homebrew, /usr/local, PATH), checks the session, reads the
CURRENT prompt (the last ❯ line; one space or nbsp after the glyph is not
input), applies the policy, sends the line literally (-l) then Enter.
Exit 0 sent · 3 no session · 4 no tmux · 5 pending text · 6 word queued.

The app's two functions are replaced by `tmuxSendLine(session:line:
skipIfQueued:)`, which execs the script from `repoRoot` with the app's
socket; the watcher path keeps its three outcomes (queued → skip, sent →
notify, else → fallback notice). Built here with the README's swiftc line
(rc=0); `restart-coordinator-swift` and `sutando-config-swift` suites OK.

tests/tmux-send-line.test.sh against a REAL tmux server on a throwaway
socket: R1 literal line + Enter delivered; R2 missing session → 3; R3 a
pane whose prompt carries text → 5 quoting it; R4 --skip-if-queued on the
queued word → 6; P1–P3 the prompt parser (last ❯ wins; nbsp-empty reads
empty; leading whitespace ignored). The model-switch script (#3898)
switches to this sender once it lands.

Stand: Echo Act IV Mini

* fix(model-switch): one lock across preflight, transaction and send; SKILL.md matches the code

qingyun-wu round 4 on #3898:
1. Overlapping invocations could interleave the settings write, the
   record commit and the live send (measured: settings haiku, record
   sonnet). A flock on `<state>/.model-switch.lock` (fd 9, held for the
   process lifetime) now spans preflight, the whole transaction and the
   send. Tests 26/27: two concurrent switches with a delayed capture —
   settings, record and the last live command agree; the sends do not
   interleave. A lock path that cannot be opened (state path is a file)
   refuses with "nothing changed" — the earlier partial-failure check
   caught the bash redirection error that replaced that message.
3. SKILL.md rewritten to the actual ordering: preflight → prior bytes
   read → record staged → pin → record committed (settings rolled back on
   commit failure) → send; defaults from the runtime descriptor.
2. The safe-host restart witness remains the owner's ask, stated as such.

27 checks; committed after the suite's exit code was read.

Stand: Echo Act IV Mini

* fix(tmux-send-line): fail closed on a failed inspection; one lock per socket+session; shim suite always runs

qingyun-wu on #3899:
1. A failed capture-pane or a missing/failing parser interpreter read as
   an empty prompt and the send went ahead. Now: `set -o pipefail`, the
   interpreter is checked before use, capture and parse are checked
   separately, and any failure refuses with exit 7. Controls S5 (capture
   fails) and S6 (interpreter fails): nothing sent.
2. Two callers could interleave payloads before either Enter. A flock on
   a per-socket+session lock file (fd 9) now spans inspection and both
   send-keys. Control S7: two overlapping callers with a delayed capture
   serialize to lit/enter/lit/enter.
3. The suite's assertions were gated on tmux living at two fixed paths.
   The shim leg (S1–S7) now runs unconditionally through the PATH shim
   the resolver finds first; the real-tmux leg (R1–R4) is the optional
   second leg and runs here.

Swift unchanged from 220e684 (built there, rc=0).

Stand: Echo Act IV Mini

* refactor(model-switch): the live half goes through the shared sender; stacked on #3899

qingyun-wu round 5 on #3898: two locks on one pane are not one lock. The
switch script no longer touches tmux itself — the preflight is the shared
sender's --dry-run (--refuse-if-pending, under its socket+session flock)
and the send is the shared sender proper, so a model command and the
app's watcher line cannot interleave. Local has-session/capture-pane/
send-keys: 0 remaining. The switch keeps its own transaction lock for
settings+record.

Test 28 (cross-sender): a switch and a watcher line through the same
sender with a delayed capture serialize to lit/enter/lit/enter.
28 checks, committed after the suite's exit code was read.

Merge order: #3899 first (this branch's base), then this.

Stand: Echo Act IV Mini

* model-switch: drop the settings.json pin — the CLI persists /model itself

Owner, 2026-09-04: "isn't the PR about changing model without restart?"
It is. The pin was the one thing in the script that touched a
launch-time path, and it duplicated a write Claude Code already makes:
measured on this host, settings.json's `model` key changed at 14:08Z,
hours into a session, with no writer in the repo — the CLI's own /model
wrote it. So the script now records the switch (previous model read
from settings.json, read-only) and types /model through the shared
sender under its lock; it never writes settings.json, and there is no
launch-time claim left to witness across a restart.

Suite rewritten to the narrowed contract: 19 checks, including
"settings.json is byte-identical after every case" (3, 19) and "the
script touches no tmux itself" (15); the race (16/17) now asserts record
== last live send; cross-sender (18) kept. SKILL.md and the catalog
entry say what the script does now. Committed after the suite's exit
code was read.

Stand: Echo Act IV Mini

* fix(model-switch): the runtime gate fails closed — an unresolved or unknown runtime refuses before any record

qingyun-wu round 6 on #3898: `core-runtime` failing or printing nothing
was read as claude, and SUTANDO_CORE_RUNTIME=bogus still wrote a record
(rc=3). Now a nonzero resolver, an empty value, or a runtime other than
claude/codex refuses with exit 4 before the record or any pane action.
Tests 20/21 pin both shapes (a bogus value, which the resolver itself
rejects; a resolver that exits 1 via a wrapper repo). 21 checks.

Stand: Echo Act IV Mini

* fix(model-switch): the send is initiation, not success — record only after the CLI accepts

qingyun-wu round 7 on #3898 (P1): on a warm core the CLI shows a confirm
dialog, the old script exited 0 and wrote the record while the model had
not changed. Now: baseline the acceptance lines on screen, send, then
wait for a NEW 'Set model to' line (pane-observe.sh, the skill's only
pane reader). The dialog is answered only under --confirm (an owner
instruction is the authorization); otherwise it is cancelled with Escape
and the script exits 6 with nothing recorded. No acceptance within
--accept-timeout (20s) is exit 8, nothing recorded. previous falls back
to the last record when settings.json carries no model key and names its
source. Controls 22-26; 26 checks. Parent control: same dialog shim, the
old script exits 0 and writes the record.

Stand: Echo Act IV Mini

* fix(model-switch): accept only the requested model, and snapshot previous before the send

qingyun-wu round 8 on #3898, two P1s. (1) previous was read after the
CLI accepted — and persisted — the new model, so a normal switch to opus
recorded previous=opus; it is now snapshotted before the send and carried
to the record. (2) the observer counted any 'Set model to' line; it now
matches the REQUESTED model's display name (family + version derived from
the alias or claude-<family>-<ver> id), so another model's acceptance in
the polling window is exit 8, nothing recorded. Headers cut to two lines.
Fixture: the shim renders the real CLI display name per accepted send,
TMUX_ACCEPT_AS forces a wrong model, TMUX_PERSIST_SETTINGS mimics the CLI
saving the pick. Controls 27-29; 29 checks. Parent controls: wrong-model
rc=0 + record; persist -> previous=opus. Head: rc=8 + none; previous=fable.

Stand: Echo Act IV Mini

* fix(model-switch): match the exact display name, and refuse when the pane cannot be read

qingyun-wu round 9 on #3898, two P1s. (1) 'Set model to Opus 5' as a bare
substring also matched 'Opus 5.1', so a request for one version could
accept its neighbour; the acceptance regex now ends at a boundary (a
versioned id matches only its version; an alias accepts the family at any
version). (2) a failed capture-pane read as count 0 with exit 0, so a later
stale line could pass as new; the observer now propagates the failure and
switch-model refuses BEFORE sending. Controls 30-33; 33 checks. Parent:
adjacent version rc=0 + record; failed baseline proceeds to send. Head:
rc=8 + none; rc=7, nothing sent.

Stand: Echo Act IV Mini
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants