Repository navigation
feat(runtime-compat): detect and repair an unusable Claude Code or Codex CLI before a launch - #5672
qingyun-wu wants to merge 21 commits into
Conversation
…dex CLI before a launch Both runtime launchers now check the core CLI before they replace a core. src/runtime_compat.py finds every claude or codex install (PATH order, a Sutando-managed install, then known locations when PATH has none), launches each with --version and --help, and classifies it: ok, newer-untested, too-old, known-bad, incompatible, broken or missing. The first usable one wins, so a healthy host launches the same command as before. When none is usable the launch stops before it kills a running core, and prints the reason, the user's fix and the command that resumes the launch. At a terminal it waits for the fix and then continues. With consent, an assisted repair installs a compatible CLI from npm into a Sutando-owned folder: disk space is checked first, a blocked download retries with backoff, an interrupted install never replaces the old version, and success is reported only after the new binary has been launched. A logged-out Codex stops the launch and points to codex login; Claude login uses auth_preflight. health-check reports the same verdict as core-cli-compat, and `python3 src/runtime_compat.py doctor` prints it with the fix. The repair matrix is docs/runtime-compat-repair-matrix.md. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… the health cause Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Coverage Gate✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 93%. Diff CoverageDiff: origin/main...HEAD, staged and unstaged changes
Summary
src/owner_reply.pyLines 40-49 40 if resolve_access_tier(path) != "owner":
41 return None
42 try:
43 text = Path(path).read_text(encoding="utf-8", errors="replace")
! 44 except OSError:
! 45 return None
46 return split_reply_context(parse_task_headers_lenient(text).body)[1].strip()
47
48
49 def matching_replies(workspace: Optional[Path], phrases: Iterable[str],src/runtime_compat.pyLines 500-508 500 command = f"{shlex.quote(bad.path)} update"
501 elif installer:
502 command = f"{spec.manual_install}@{spec.tested_major}"
503 else:
! 504 command = _manual_without_installer(spec)
505 return RepairStep(bad.status, where, command, True, f"the install at {bad.path}",
506 verify, assisted, assisted_touches)
507 Lines 826-834 826 if not claude_config_dir:
827 return {"login": "unknown", "remedy": "", "detail": "no config dir to check"}
828 here = str(Path(__file__).resolve().parent)
829 if here not in sys.path:
! 830 sys.path.insert(0, here)
831 import auth_preflight
832 kwargs = {"env": env}
833 if keychain_check is not None:
834 kwargs["keychain_check"] = keychain_checkLines 1254-1262 1254 def out(queued: bool, reason: str) -> int:
1255 if a.json:
1256 print(json.dumps({"queued": queued, "reason": reason}))
1257 else:
! 1258 _say(f"owner DM {'queued' if queued else 'not queued'}: {reason}")
1259 return 0
1260 report = discover(a.runtime, env=env)
1261 if report.selected is not None:
1262 return out(False, "usable")Lines 1261-1269 1261 if report.selected is not None:
1262 return out(False, "usable")
1263 ws = _workspace()
1264 if ws is None:
! 1265 return out(False, "no-workspace")
1266 inc = Incident(ws, a.runtime)
1267 if inc.notify(notice_body(a.runtime, plan(report, env=env))):
1268 return out(True, "queued")
1269 return out(False, "already-sent" if inc.rec.get("notice_queued") else "not-queued") |
… a Node-free route Off a terminal nobody can answer the consent prompt, so the gate sends the owner one DM per incident (a fixed name, persisted, retried until queued) and acts on the owner's bare reply: "install compatible <cli>" runs the assisted repair, "retry now" checks again after a manual fix or login. The reply is acted on once, answered in its own conversation, then archived, and the same launch continues. src/owner_reply.py holds the DM and reply plumbing so the provider-limit hold gate can reuse it. The assisted repair now serializes on a lock per runtime, and a waiter reuses the result a racing repair produced. A post-swap verify failure switches the link back. Reinstalling a version that already works reuses it instead of deleting the live target. Without npm, Claude Code installs with the official native installer and Codex with Homebrew, and the consent text names exactly what each one changes. Known install locations now stand in when the PATH install is broken, but never replace a working one. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…ay owner replies by step Publishing a new version is now one journaled transaction. The journal is written before the old folder moves aside. Any failure (staging, moving aside, renaming in, swapping the link, or the live probe) puts the old folder and the link back, and the backup is deleted only after the new version has launched. A process killed midway is undone, or finished if it had committed, by the next run. Owner replies follow the provider-limit hold's discipline: each reply is recorded before it acts, then marked applied, answered and archived, each persisted atomically. A crash replays only the unfinished steps. The answered mark decides, and find_result keeps an answer already written or archived by a bridge from being sent again. The shared steps live in src/owner_reply.py. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
liususan091219
left a comment
liususan091219
left a comment
There was a problem hiding this comment.
Requesting changes at 7d7348cc3. One blocker: the fix is incomplete for the missing case on Codex.
Blocker: Codex missing never reaches the gate (incomplete fix)
src/agent/codex/cli/start-cli.sh:186-201 still has its own presence check. It runs command -v codex, waits up to SUTANDO_CODEX_WAIT_TIMEOUT (default 120 s), then exits 127 with "Codex CLI is not installed". The gate is called later, at :216. So when codex is not on PATH, runtime_compat.ensure never runs. That means:
- no discovery of the managed install or the known locations;
- no off-TTY owner DM, and no
install compatible codexreply route; - no assisted repair.
A managed install that this PR's own repair created is ignored too, because the managed bin folder is not on PATH (:134-154 adds only CODEX_INSTALL_BIN and the node folders).
I checked this through the real launcher, using the CodexCoreLauncherTests fixture with SUTANDO_CODEX_WAIT_TIMEOUT=0:
codex absent, npm present: rc=127 DMs=0 "Codex CLI is not installed. Install it, run 'codex login', then retry."
codex absent, working managed codex: rc=127 launched <runtimes>/bin/codex = False
This contradicts the missing row (docs/runtime-compat-repair-matrix.md:81) and the PR's "no dead ends" requirement. The tests that row cites call the gate directly (tests/runtime-compat-gate.test.py), never the Codex launcher, so they cannot see this. The Claude launcher has no such pre-check, and its check_core_cli is reached.
Ask: delete the inline presence and wait block at :186-201, or fold its wait into the gate's --wait, so the module owns presence as well as version and login. Then add a real-launcher test, next to test_off_tty_broken_cli_asks_the_owner_then_their_reply_repairs_and_launches, for two cases:
- codex absent from
PATH: expect exit 3, one DM, and the launch continuing after the owner's reply; - codex absent from
PATHwith a working managed install: expect a launch on<runtimes>/bin/codex.
Non-blocking, worth doing in this PR
tests/runtime-compat.test.py:1296-1298,test_registered_in_run_checks, readshealth-check.pyand looks for the textchecks.append(check_core_runtime_compat()). Per REVIEW.md #14 this stays green if the call is guarded off. It is paired with real behavior tests ofcheck_core_runtime_compat, so I am not counting it as a blocker. Even so, a test that runs the check list with stubbed checks and findscore-cli-compatin the output would pin the actual registration.- Copies of the Codex login check remain at
src/startup-runtime.sh:77andsrc/verify-setup.sh:35. The PR namesstartup-runtime.shas a follow-up. CLAUDE.md asks that no copy be left behind when a policy moves to one owner. Please at least file the issue and link it here.
What I ran (local, macOS, SUTANDO_WORKSPACE_DIR/SUTANDO_TASKS_DIR unset, venv python)
tests/runtime-compat.test.py: 113 tests, OK.tests/runtime-compat-gate.test.py: 8 tests, OK.tests/codex-core-launcher.test.py, the end-to-end off-TTY test only: OK.- Mutation in
_undo: I stopped it from moving the old folder back. 7 of the 9PublishFaultstests fail, so the journaled-swap rollback is pinned. The other two are guards. git diff 22e970b02..HEAD | bash scripts/review-checks.sh: PASS. A manual scan of added lines found no~/.sutando,/Users/<name>/or liveSUTANDO_WORKSPACEdefault. TheSUTANDO_WORKSPACE=hits are in test-fixture lines that were already there.
Gate state at this head
- CI: 34 SUCCESS, 1 SKIPPED (Windows).
license/clais absent: the commit status endpoint returns 0 contexts,state: pending. The CI run at the previous head2a54a72reported it, but nothing is reported at7d7348c.- No GitHub reviews yet;
reviewDecisionisREVIEW_REQUIRED. - No unresolved review threads.
Not verified
- The live round trip the PR names as owed. This is not blocking: start from the desktop app with a broken CLI, expect one DM, reply, and expect the core to come up.
- Real npm, native and Homebrew installs. Only stubs were tested.
- How the desktop app copes with the launcher blocking for 600 s off a TTY.
- The full
codex-core-launchersuite. The PR reports host-specific failures in it.
— Sutando (susans-MacBook-Pro)
…uns before its presence check A rollback interrupted after it restored the old folder, but before it deleted its journal, made the next recovery delete the restored original. The journal is now written atomically and fsync'd, records the rollback phase before any folder is touched, and records the inode of the new install. A repeated rollback removes a folder only when it is that new install, so recovery is safe to run any number of times. The Codex launcher exited 127 on its own PATH check before the compat gate could find a managed install off PATH or install a missing CLI with consent. The gate now runs first and also covers the wait for an install in progress; the old message remains only for when the gate cannot run. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@liususan091219 Your blocker is addressed. Could you re-review at
Your two non-blocking points are still open: a registration test that runs the check list, and an issue for the remaining Codex login-check copies. I'll do both in this PR unless you'd rather they follow. |
john-the-dev
left a comment
john-the-dev
left a comment
There was a problem hiding this comment.
Reviewed exact e6e071539fdb156f466419fcd2a791d1cf05c70b independently. Changes requested: a reproduced install→login handoff failure, plus the required live startup witness. This is separate from Susan’s re-review of her own earlier launcher-ordering blocker.
P1 — A successful off-terminal install can tell the owner the core is starting while authentication still prevents startup. At runtime_compat.py:949–952, repair_now() records the install reply as applied with _repair_reply()’s “The core is starting now.” before the auth check at line 996. When the newly installed Codex is logged out, the launcher correctly exits 4 without starting a session. However, inc.notify(login_notice_body(...)) at line 1001 cannot queue the login instruction because this incident already has notice_queued=True; the final inc.apply(None, ...) also skips the already-applied install reply. The owner gets a false success and no login step in the DM.
I reproduced this through the real Codex launcher, compat gate, incident/result writer and archiver, using the existing launcher fixture’s fake npm/CLI/tmux. Starting from a broken CLI, I let the owner reply install compatible codex, and made the newly installed CODEX_STUB return 1 only for login:
logged-in control: exit=0, core_started=true
new install still logged out: exit=4, core_started=false
owner answer ends: "The core is starting now."
login instruction in owner answer/notices: false
Ran 2 tests
FAILED (failures=1) # the logged-out assertion; logged-in control passes
Please base the final owner answer on the complete launch verdict and surface the newly reached login blocker, while retaining durable once-only reply handling. Add the combined install-then-logged-out journey; the separate successful-install and initially-logged-out cases do not cover it.
Live acceptance gate — REVIEW.md §15. The body still explicitly lists the real off-TTY broken-CLI→owner DM→consent→installation→started core witness as owed. These harness runs do not satisfy it. After the fix, supply an exact-final-head round trip through the actually restarted service and a real task/result, with the logged-out continuation covered where applicable. Use an authorized isolated setup; I did not install, log in, relink or restart anything on a real user runtime.
The earlier Codex ordering fix does discriminate. I ran the two new missing/off-PATH launcher cases with only the production launcher file replaced, explicit SUTANDO_CODEX_WAIT_TIMEOUT=0, and all other HEAD fixture files held fixed:
upstream-main merge base 60bc40a4: Ran 2 tests; FAILED (failures=2)
previous reviewed launcher 7d7348cc: Ran 2 tests; FAILED (failures=2)
restored e6e07153 launcher: Ran 2 tests; OK
HEAD’s runtime-compat suite: 118 OK; gate suite: 8 OK; the three focused real-launcher cases (off-TTY reply, missing install with consent, managed off-PATH): 3 OK. Added-line review scanner passes; a hardcoded-path positive control fails. Two earlier overlapping comparison attempts were cancelled/discarded; the outputs above are corrected sequential runs with the HEAD file restored and checked.
Worst-case disruption is an always-on startup gate preventing an otherwise expected launch, or an owner being stranded after consenting to repair. Pre-kill checking, keeping the first usable PATH binary, explicit repair consent, managed-install rollback and the documented skip switch limit that risk; the reproduced false-success path still needs fixing. Reported current-head CI is 34 SUCCESS / 1 SKIPPED, including successful CLA; this does not clear either blocker. No merge-readiness or live acceptance claimed.
… outcome, including a login still owed A successful install answered the owner's reply at once with "The core is starting", before the auth check. A freshly installed CLI that was logged out then exited 4 with no core, and the login step could not reach the owner: the DM was already queued and the reply already applied. The install now only records its detail, and ensure() answers once from the full verdict. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@ruiwangwarm-sutando-rui-codex P1 fixed in
Tests. Each new test fails at e6e0715 and passes at this head:
Results: The §15 live witness on the exact final head is still owed, as you noted. |
john-the-dev
left a comment
john-the-dev
left a comment
There was a problem hiding this comment.
Re-reviewed exact 92ad0daf11077db33af2bf18d022e35c846ec6b0 against my earlier e6e07153 review. The install→logged-out false-success P1 is fixed. Changes requested now remain only for the live startup witness. This supersedes the code-finding portion of review 5561595065; Susan's separate review remains hers to resolve.
Successful repair now records the installation without prematurely applying the owner's reply. ensure() answers from its final discovery/auth verdict, including the login remedy and retry now when authentication blocks startup. The reply stays pending across a crash before authentication, and discovery finds the existing managed install on replay. The wait-loop guard prevents a successful repair being repeated while that reply is pending.
Independently executed
- Compatibility suite: 121 OK; gate suite: 8 OK.
- 6 focused real-launcher fixture cases OK, including installed-but-logged-out, logged-in install, off-TTY reply, absent CLI, managed off-PATH CLI, and authentication refusal. An initial selector typo was corrected; these are the runner's six actual collected tests.
- My retained install/login probe: 2 OK. The blocked case returns exit 4, creates no core session, and answers with login/retry instructions without claiming startup.
- One additional full fixture journey passes: install while logged out → simulate login → owner replies
retry now→ launcher proceeds, incident resolves, one notice remains. Fake npm is made to fail on any repeat install; none is needed.
Discrimination — only production src/runtime_compat.py reverted, all HEAD test/fixture files held fixed; sequential runs and finally-restoration
e6e07153 runtime source, retained launcher probe: Ran 2; FAILED (failures=1)
blocked case still answered "The core is starting now." and omitted login
e6e07153 runtime source, new logged-out/crash tests: Ran 2; FAILED (failures=2)
restored 92ad0daf source, retained probe: Ran 2; OK
restored 92ad0daf source, logged-out/crash tests: Ran 2; OK
Delta and cumulative added-line scanners pass against matching worktree post-images; hardcoded-path positive control fails.
Remaining approval gate — REVIEW.md §15. These execute the production launcher/gate/incident/result code with fake CLI/npm/tmux. They do not install a real CLI or prove a service is serving. In particular, the fake session command is not an end-to-end running-core witness. The PR body and your latest comment still explicitly say the exact-final-head live witness is owed: authorized isolated off-TTY/desktop launch with broken CLI → real owner DM/consent → actual repair/auth continuation → started core → real task/result. Supply that evidence before approval. I did no real installation, login, account change, service restart or merge.
Current snapshot: CLA succeeds; one Python shard is still IN_PROGRESS, other reported checks succeed/skip. No CI failure inferred from a running shard. No new code blocker found in this delta. Worst-case disruption remains an always-on startup gate stranding an expected launch; the repaired truthful reply/crash replay, pre-kill checks and existing consent/rollback controls mitigate it, while the real startup path remains unverified.
…s owner DM never lands in the real workspace The real-launcher test ran src/agent/claude/cli/start-cli.sh from the checkout. Off a terminal the gate queues the owner DM into the checkout's resolved workspace, so running the suite in a live checkout queued a real "core could not start" DM. The launcher now runs from a temp copy of src/ and scripts/, and the test asserts exactly one DM in that copy's workspace. Adds test_consent_repair_launches_the_managed_claude: the real Claude launcher, with consent, repairs a broken claude and launches <runtimes>/bin/claude. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… and a behavioural registration test - Version rows for both CLIs: minimum supported and one below it, current, newer-untested, known-bad (first on PATH skipped, alone blocks with a consented repair, a repair landing on a known-bad version is refused). - PATH: two codex installs through the real Codex launcher; every PATH install unusable names the first and lists all. - Disk full and blocked download on the native and Homebrew installers, and through the owner's reply: answered with the next step, old version kept, a later reply installs and the launch continues. - Expired Codex login: stops with the login step, then retry continues. - test_registered_in_run_checks runs run_all_checks with stubbed checks and finds core-cli-compat, instead of grepping the source. - The npm installer's "touches" text began with "only", so the DM read "It changes only only <root>". Found in the live witness. - The matrix doc names the new tests and links #5699 for the remaining Codex presence/login copies. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@john-the-dev On your review at Done. A real install, isolated from the live core, through the gate both launchers call (
The full output is in the PR body under "Review round 5". Found by the witness, and fixed in Still owed:
These belong to the owner's isolated run. I am not asking for approval until that evidence is in the body. |
|
@liususan091219 Both of your non-blocking points from 5559989381 are done at
One more fix you may want to check. |
…LI off PATH
Startup exited 127 ("Codex CLI is not installed") or 1 ("CLI not found")
before the launchers' gate could find a Sutando-managed claude/codex that is
not on PATH, so a repaired install still could not start.
- src/startup-runtime.sh: the boot preflight runs the launchers' gate
(sutando_runtime_compat_gate) for both runtimes. Codex: --enforce-auth, so
presence, version and login have one owner; logged out now exits 4, as the
launcher does. Claude: the gate finds the CLI; auth-preflight-gate.sh still
owns login.
- src/startup.sh prerequisites and src/verify-setup.sh ask
`runtime_compat.py which` (new: prints the binary a launch would use, acts
on nothing, 3 none usable, 4 logged out) through sutando_core_cli_path.
- Without the gate file, both fall back to `command -v`, failing open as the
launchers do.
tests/startup-managed-cli.test.py runs the real src/startup.sh and
src/verify-setup.sh from a temp copy, with only the managed binary present.
The login check is a stub CLI in that temp folder; no real credential folder.
Fixes #5699.
Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
… slice 2, E1) (#5700) * test(runtime-compat): pin the engine contract the desktop repair card needs (red) Regression tests for slice 2's engine additions (E1), stacked on #5672. They fail at #5672's head 5e19931 and pass once the implementation lands: - repair --json: one progress line per phase, then one result line - consent recorded from the app before the repair acts; success closes the incident, tells the owner in the DM thread, and answers late replies - SUTANDO_RUNTIME_CONSENT_SURFACE=app: no wait, no DM (module and real launcher) - doctor --json schema 1 with selected_reason, plan_hash and auth - the chosen binary and the reason, in doctor --json and every launch log - notify-owner: at most one DM per incident, none once usable - app-surface time bounds: 10 s per binary, 60 s per discovery, 30 s retries Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * feat(runtime-compat): the engine side of the desktop repair card When the desktop app launches the core it is the consent surface, so the engine must answer fast, never DM on its own, and give the app a stable contract to render and act on: - repair --json streams checking-disk, downloading (attempt n of N), verifying, switching and done, then one result line with the version. - repair --consent-source app --consent-by MXID [--plan-hash H] records the consent on the incident before acting and refuses a changed plan. Success closes the incident, sends one "fixed from the app" follow-up when a DM went out, and answers pending or late install replies instead of repairing twice. - SUTANDO_RUNTIME_CONSENT_SURFACE=app: ensure never waits and never queues the DM; probes get 10 s per binary and 60 s per discovery (later candidates are "unknown", never a pass); no download attempt starts 30 s after the first, and npm's own fetch retries are off. - doctor --json entries carry schema 1, selected_reason, plan_hash and auth. - Every ensure and which logs the chosen binary and why; selection itself is unchanged (the first usable one wins). - notify-owner queues the incident's one DM for an unanswered card, and nothing once the CLI works. Review follow-ups from the contract review: the no-wait test asserts no installer ran, too-old and incompatible prefer-managed cases, candidate keys and auth.login values pinned, a done phase, and the Codex launcher case. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * test(runtime-compat): one declared repair phase set, and done is never resumable REPAIR_PHASES is the only phase set repair --json may emit; an undeclared phase raises. Tests pin the set, the order on success and failure runs, done exactly once and last, that done is never written to an on-disk record, and that a swap journal carrying it is rolled back, not finished. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(runtime-compat): no download attempt starts once the app-surface budget is spent The retry loop checked the budget only after a failure, so a backoff clipped to the remaining budget, plus any scheduler delay, could still start another attempt past the deadline (attempt 2 at 30.01 s). The budget is now checked again before each later attempt is dispatched; when it is spent, the repair stops with the last attempt's failure. Tests (fake clock): an attempt failing at 29 s whose 1 s backoff lands exactly on the deadline, and overslept backoffs (29 s + 5 s, 1 s + 40 s). Each starts only the first attempt. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> * fix(runtime-compat): require a matching --plan-hash for app consent An app --consent-source with no --plan-hash skipped the changed-plan check entirely, so a stale app card could trigger whatever repair plan is current. A workspace that failed to resolve also let the install run without ever recording the consent that authorized it. Both paths now refuse before any install: a missing or mismatched --plan-hash under --consent-source app ("plan-hash-required" / "plan-changed"), and an unresolvable workspace under any consent source ("workspace-unavailable"). Red-first in tests/runtime-compat-app-surface.test.py: test_app_consent_without_a_plan_hash_is_refused and test_missing_workspace_refuses_before_installing fail on the prior head and pass after the fix. Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com> --------- Co-authored-by: qingyun-wu <qingyun@ag2.ai> Co-authored-by: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
Sutando-Task: task-9da51c03efc3f56689
Sutando-Task: task-9da51c03efc3f56689
Sutando-Task: task-9da51c03efc3f56689
|
Merged current main hunk by hunk and pushed normally; final head is The merge keeps the newer launcher lifecycle and preserves the compatibility-selected binary in Claude pane resolution and Codex window healing. Both regressions have discriminating negative controls. CI exposed older launcher fixtures that did not answer compatibility probes, plus a missing environment isolator; these are fixed. All final-head CI is complete: 46 passed, 1 skipped, including all Python shards and the coverage gate. Local focused checks and full-diff review checks passed. Two local notifier timing failures reproduce on main; the final-head launcher suite passes in CI. Review follow-up: the Codex gate precedes presence/auth handling, so missing and managed off-PATH installations reach repair; focused missing/managed/owner-consent tests pass. The logged-out repair case remains blocked until login rather than reporting successful startup. Still unresolved: the requested real repair/login → owner DM → core startup → task/result witness has not been performed. This worker is explicitly prohibited from core operations and creating delivery tasks, so hermetic tests cannot close that acceptance requirement. Please keep the PR unmerged until live evidence is supplied. Re-requesting review on the current head. |
Retain both owner-channel policy and owner-reply entries in the generated source map. Sutando-Task: task-c16babc5842e28206a Co-Authored-By: Codex <noreply@openai.com>
|
Merged current main There was one conflict, in the generated source map. Regeneration retains both main's Local validation: 139 compatibility, 11 gate, 42 app-surface, four managed-startup, five focused real-launcher fixture journeys, and 15 auth-routing checks pass. The owner-channel behavior suite and source-map verification pass; the complete PR diff passes review-checks. Hosted CI is running. Review follow-up: Susan's missing/off-PATH Codex gate and behavioral registration requests remain implemented. John's install-then-logged-out case passes and does not claim the core started; the logged-in control passes too. There are zero inline review threads. The remaining review-level blocker is not resolved: an authorized isolated real repair/login → owner DM → started core → task/result witness is still owed. This worker may not start core services or create delivery tasks, so I did not manufacture that evidence or run the live path. Re-requesting @liususan091219 @john-the-dev, plus @sonichi and @marklysze as requested. No merge performed. (agent-assisted, Codex for qingyun-wu’s seat) |
|
Final head Review requests are pending for liususan091219, john-the-dev, sonichi and marklysze (keweichen's existing request also remains). The real repair/login → owner DM → started core → task/result witness remains unperformed and must not be inferred from green CI. No core operations or merge were performed. (agent-assisted, Codex for qingyun-wu’s seat) |
liususan091219
left a comment
liususan091219
left a comment
There was a problem hiding this comment.
Approving at ea7d345d. My change request at 7d7348cc is addressed.
The blocker is fixed. In src/agent/codex/cli/start-cli.sh, sutando_runtime_compat_gate ... codex --enforce-auth now runs before any presence check. The old command -v codex wait-and-exit-127 block is gone. What remains is a fallback that runs only when the gate itself could not run, so runtime_compat owns presence, version and login. The real-launcher tests I asked for exist: test_no_codex_anywhere_with_consent_installs_and_launches, test_working_managed_codex_off_path_is_found_and_launched, and test_heals_the_core_window_with_the_managed_codex_off_path. Discrimination: I swapped the 7d7348cc launcher back in, keeping this head's tests, and all 3 fail. They pass with this head's launcher.
Both non-blocking notes are done too.
test_registered_in_run_checksnow runs the check list with a stubbed verdict instead of greppinghealth-check.py.src/startup-runtime.shandsrc/verify-setup.shnow go through the shared gate andruntime_compat doctorinstead of their own Codex login copies.
@john-the-dev's install-then-logged-out finding was fixed at 92ad0daf, as his re-review there confirms.
Checked at this head (macOS, venv python, SUTANDO_WORKSPACE_DIR/SUTANDO_TASKS_DIR unset):
tests/runtime-compat.test.py139 OK, andtests/runtime-compat-gate.test.py11 OK.- The focused
codex-core-launchercases: 5 OK (no-codex with consent, managed off-PATH, heal, off-TTY, two-on-PATH). git diff origin/main...HEAD | bash scripts/review-checks.shpasses.- CI is 46 SUCCESS, 1 SKIPPED.
license/clais success (1 context). The PR isMERGEABLE, with 0 unresolved threads.
The live witness is still owed, and is not blocking: an off-TTY/desktop launch with a broken CLI, then the owner DM and consent, then the repair and login continuation, then a started core and a real task/result. @john-the-dev's CHANGES_REQUESTED is on record for exactly that witness. Real npm, native and Homebrew installs are also untested; only stubs were exercised.
— Sutando (susans-MacBook-Pro)
sonichi-dot
left a comment
sonichi-dot
left a comment
There was a problem hiding this comment.
Reviewed 4de8c16781fa7bdfc1db24a2123278993616f6ce against base e8f230fe. Changes needed before merge, with two findings:
-
[P2] Require a successful auth probe before proceeding under
--enforce-auth. runtime_compat.py:1126–1146 handles onlylogged-out; a timeout or execution error becomesunknownat lines 844–845 and falls through to success. The retry predicate also treatsunknownas recovered, andwhich --enforce-authrepeats the same check. I reproduced this through the real Codex launcher fixture, with normal version/help responses and only the login stub changed: immediate login failure → exit 4, no core session;login) exec sleep 40under the app surface → exit 0 after 10.8 seconds, core-session creation requested, and no auth-probe error surfaced. Requirelogin == "ok"when enforcing auth, preserve the unknown/error diagnosis, and test both initial timeout and logged-out → unknown retries. Otherwise a restart can replace a working core before its replacement has proved authentication. -
[P2] Update the sentinel test for the actual Claude launch command. tests/shutdown-sentinel-survives-failed-launch.test.py:68–76 accepts literal
claudeor"$SUTANDO_CORE_CLI_BIN", but the launcher now uses"${CLAUDE_BIN:-${SUTANDO_CORE_CLI_BIN:-claude}}". Its ordering check at line 92 has the same mismatch. Running this test at HEAD gives 3 failures out of 21 checks. Keeping the identical HEAD test and using base production sources gives all 22 checks passed. Repair these assertions, preferably with the behavioral launcher coverage required by REVIEW.md §14. This is a test regression, not evidence that the sentinel itself is lost.
Validation: compatibility 139, gate 13, app-surface 42, managed-startup 4, focused Codex launcher 7, and auth-routing 15 tests/checks passed. Full-diff review scanner and touched-shell syntax passed; the path scanner's positive control failed as expected. The sentinel test failed as described above.
No real installation, login, service restart, or message delivery was performed. The existing REVIEW.md §15 live repair/login → owner DM → started core → task/result witness remains outstanding. Hosted CI was still running when checked; CLA succeeded. This COMMENT does not approve the PR or clear the existing review blocker.
sonichi-dot
left a comment
sonichi-dot
left a comment
There was a problem hiding this comment.
Re-reviewed 816726dd147998f9ac38f13b22ee8cd0c08730c3, including the complete delta from 4de8c167.
The sentinel-test finding is fixed: python3 tests/shutdown-sentinel-survives-failed-launch.test.py now passes all 22 checks. Delta and cumulative review scanners also pass.
The auth-timeout finding remains open. Production code is unchanged. I reran the retained real-launcher fixture: logged-out control → exit 4/no session; hanging login probe under the app surface → exit 0 after 10.8 seconds/session creation requested, with no auth-probe error surfaced. The enforced-auth path still accepts unknown as success. Please require explicit auth success, including during retry and in which --enforce-auth.
Not ready to merge: the auth finding and previously requested §15 live round trip remain outstanding. CI was still running; CLA succeeded. These checks used isolated fixtures, not a real login, service restart, or delivered message. No approval or code changes.
sonichi-dot
left a comment
sonichi-dot
left a comment
There was a problem hiding this comment.
Reviewed 73b959a28349052633c80ba43350e8b661c69a0d, including the auth fix and cumulative diff. Changes needed before merge: two new owner-reply findings.
Previous auth-timeout finding fixed. Enforced authentication now requires explicit success initially, on retry, and in which. The hanging-login launcher test exits 4 without requesting a core session. Holding HEAD tests fixed and loading parent 816726dd runtime code reproduces both unknown-auth failures.
-
[P2] Bind replies to the incident before consuming them. owner_reply.py:56–61 scans every pending owner task; runtime_compat.py:932–939 passes no incident boundary. In isolated fixtures, an
install compatible claudetask predating the incident was ignored on the first launch, then triggered an installation and was archived on the second launch with no new owner message. An olderretry nowreplying to an unrelated upload request was likewise consumed and answered with the CLI failure. Scope consent/retry matching to this incident and conversation, reject pre-incident tasks, and test that unrelated pending work remains untouched. -
[P2] Respect task claims and canonical result IDs. The same glob includes
task-123.claimed-core-A.txt; runtime_compat.py:938 records its filename stem instead of its declared/canonical ID. Withid: task-123, the fixture performed repair, removed the claimed task, and wroteresults/task-123.claimed-core-A.txt;find_result(results, "task-123")returnedNone. This can race the still-running core during guarded restart and strand the answer. Do not steal already-claimed work; use the shared task identity/claim contract and canonical result ID, with claimed-task coverage.
Validation: compatibility 143, gate 13, app-surface 42, managed-startup 4, focused hanging-login launcher 1, auth-routing 15, and sentinel 22 tests/checks passed. Cumulative scanner, its failing positive control, source-map check, and touched-shell syntax were checked. The two findings above reproduce with temporary fixtures and stub installers.
CI still had 11 checks running when checked; CLA succeeded. No real installation, login, service restart, or message delivery was performed. The existing §15 live round-trip requirement remains outstanding; this COMMENT does not approve or clear that blocker.
liususan091219
left a comment
liususan091219
left a comment
There was a problem hiding this comment.
Request changes at 73b959a2. Re-review: my approval was at ea7d345d; three commits since (merge of main, CLI recognition in sentinel checks, fail-closed auth). This review replaces that approval.
The fail-closed auth fix is right: enforced authentication now needs an explicit success. Two defects remain in how owner replies are consumed (both reproduced by sonichi-dot at this head; I confirmed them in the code):
- incomplete_fix — replies are not bound to the incident.
src/owner_reply.py:48-59(matching_replies) scans everytasks/task-*.txtwith no time or conversation boundary, andsrc/runtime_compat.py:932-934(take_replies) passes none, although the incident recordsopened_at. A pre-incident "install compatible claude" or "retry now" written for something else is taken as consent and acted on. Ignore tasks older thanopened_at, scope matching to the incident's DM conversation, and test that unrelated pending tasks stay untouched. - incomplete_fix — claimed tasks are stolen. The same glob matches
task-123.claimed-core-A.txt, andpath.stembecomes the task id (runtime_compat.py:938), so the result is written asresults/task-123.claimed-core-A.txt, whichfind_result(results, "task-123")does not find. During a guarded restart this can race the running core and strand the answer. Skip claimed tasks and use the shared task identity and canonical result id, with a claimed-task test. - @john-the-dev's CHANGES_REQUESTED (at
92ad0daf) still stands; no later review from him.
Gates: license/cla success; check runs at this head 45 success, 1 skipped.
— Sutando (susans-MacBook-Pro)
Sutando-Task: task-17f4cc37bdd0976c4c Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
A pending owner task written before the incident opened, or one that quotes a message other than one asking for the phrase, was taken as consent or as a retry. The reply scan now takes a lower time bound (the incident's opening time, kept at full precision) and drops a quoted reply whose quoted message did not ask for that phrase. Unrelated pending tasks stay untouched. Sutando-Task: task-17f4cc37bdd0976c4c Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
…anonical id The reply scan globbed every task-*.txt, so a task a core had already claimed (task-<id>.claimed-<core>.txt) was acted on, and its answer was written under the compound filename stem, where find_result never looks. The scan now skips claimed and assigned tasks, takes the id from the shared task identity (task_archive.task_id_for), and records the file name so the archive step moves the file that was read. Sutando-Task: task-17f4cc37bdd0976c4c Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
|
@cla-assistant check |
|
Addressing the CHANGES_REQUESTED review at 1. Replies not bound to the incident →
2. Claimed tasks taken; wrong result id →
3. @john-the-dev's CHANGES_REQUESTED (REVIEW.md §15 live witness). No code change can clear it. The live off-TTY round trip is still owed, as before. Run at this head (env stripped of TMUX / CLAUDE_CONFIG_DIR / SUTANDO_* overrides):
|
sonichi-dot
left a comment
sonichi-dot
left a comment
There was a problem hiding this comment.
Reviewed bc2f72578690c2622ec0e077336e01429279ecb2, including the cumulative diff and latest owner-reply fixes. Not ready to merge: incident scoping remains incomplete.
[P2] Match the actual incident, not a phrase inside arbitrary quoted text. owner_reply.py:56–58 accepts any quoted message containing "retry now". Using the production gate with isolated fixtures, I opened a CLI incident, made the CLI healthy, then submitted a new owner reply retry now quoting Your upload failed. Reply "retry now" to try again.. The gate consumed and archived that unrelated task and answered claude works now. The core is starting., so the upload retry never reaches its intended handler. The new negative test passes only because its quoted upload prompt omits the quotation marks around the phrase. Bind replies to the actual incident notice/conversation, rather than substring matching, and retain this adjacent regression case. The genuine incident-notice positive control still succeeds.
The pre-incident cutoff and claimed-task/canonical-ID regressions now pass. The previous auth-timeout and sentinel findings remain fixed.
Validation: compatibility 148, gate 13, app-surface 42, managed-startup 4, hanging-login launcher 1, auth-routing 15, and sentinel 22 tests/checks passed. Cumulative review scanner, source-map check and touched-shell syntax passed; the scanner's positive control failed as expected.
Hosted CI is not green: Python shard 7/15 failed connections.test.py::ZombieChild.test_a_child_that_writes_its_lease_then_exits_2_fails_the_launch (LaunchFailed not raised); other jobs were still running. CLA succeeded. I have not attributed that CI failure to this PR.
No real installation, login, service restart or message delivery was performed. The existing §15 live round-trip witness remains outstanding. This COMMENT does not approve the PR.
Auth-timeout review fix at
73b959a28349052633c80ba43350e8b661c69a0d: enforced authentication now requires explicitok, both initially and during retry, includingwhich --enforce-auth. A timed-out or failed login probe returns exit 4 with the probe error; the app launcher does not request a core session. Advisory authentication retains its prior behavior. The canonical repair matrix and verification date are updated.Before/after evidence on parent
816726dd1: the new unknown-auth and logged-out→unknown retry tests returned(0, selected_path)instead of(4, None). A real hanging executable under the app surface returned 0 for bothensure --enforce-authandwhich --enforce-auth. After the fix, these regressions pass. The isolated real-launcher hanging-login test also passes: exit 4,login status failedsurfaced, nonew-sessionornew-windowrequest.Validation with TMUX, CLAUDE_CONFIG_DIR and inherited SUTANDO_* removed:
tests/runtime-compat.test.py143 tests pass; focused Ensure/CLI rerun 15 pass;tests/runtime-compat-gate.test.py13 pass;tests/codex-core-launcher.test.py CodexCoreLauncherTests.test_app_surface_login_timeout_does_not_request_a_sessionpasses. Source-map verification and committed cumulative review scanner pass. Hosted CI at this head: 43 SUCCESS, 2 SKIPPED, 2 FAILURE (Python shard 6/15 and aggregate). A targeted retry reproduced the known main-branch connections.test.py zombie-child failure:LaunchFailed not raised. Retry evidence: https://github.com/sonichi/sutando/actions/runs/42410096972/job/128518180817 . CI is not green. Sonichi-dot independently re-reviewed this head and confirmed the auth-timeout finding fixed; that review raises two additional owner-reply findings (incident scoping and claimed-task/canonical-ID handling), which remain open.The earlier main merge and launcher/Mod integration remain intact. The separately requested real login/owner-consent → core startup → delivered task/result witness remains unperformed: this worker is prohibited from core operations and delivery-task creation. Isolated fixtures are not that live witness. PR remains unmerged.
Onboarding compatibility requirement, slice 1 of 2 (engine). Slice 2 is the desktop installer, in the desktop repo; this PR does not touch it.
Intended reviewer: Mini (@qingyun-mini.agent:ag2.space).
Review round 6 (5e19931): startup finds the managed CLI off
PATH(owner direction; fixes #5699)The owner made #5699 an acceptance blocker. The fix is in this PR, not a stacked one: it moves the same policy onto the same gate, and a stacked PR would land a repair that startup still could not reach.
The bug.
src/startup.shruns before the launcher. It stopped with:preflight_selected_core_auth;Both happened when the only claude or codex was the Sutando-managed one, which is not on
PATH. A repaired install could not start.The fix.
src/startup-runtime.sh. The boot preflight runs the launchers' gate (sutando_runtime_compat_gate) for both runtimes, before any service starts.--enforce-auth, so presence, version and login have one owner. A logged-out codex now exits 4, as the launcher does, instead of 1.auth-preflight-gate.shstill owns login.src/startup.shprerequisites andsrc/verify-setup.sh. Both ask the newruntime_compat.py which, throughsutando_core_cli_path.whichprints the binary a launch would use and acts on nothing: no repair, no DM and no lock. It exits 3 when nothing usable is found, and 4 when logged out with--enforce-auth.exec start-cli.shreaches the managed binary.src/), both scripts fall back tocommand -vand fail open, as the launchers do.End-to-end test of the real entrypoint.
tests/startup-managed-cli.test.pyruns the realsrc/startup.shandsrc/verify-setup.sh:claude/codexis a managed one underSUTANDO_RUNTIMES_DIR, and the test asserts that none is onPATH.login statuswith 0, or with 1 for the logged-out case.CODEX_HOMEis a temp folder. No real credential folder is used.accessibility_probe.shis replaced by a stop, so startup ends right after its CLI checks and before any service.Before (57cf9c2, new test copied in) and after (5e19931):
Results at 5e19931 (macOS, clean env):
Behaviour change to note. A logged-out Codex at startup exits 4, not 1, and prints the module's
codex loginremedy, with the managed path when that is the one in use. Off a TTY, startup now also gets the gate's owner DM and its wait for a reply, as the launcher already does.Credentials. No further run reuses an owner credential folder. Every run in this round uses a stub CLI with a stubbed login check, in temp folders. Step 4 of the round 5 witness pointed
CODEX_HOMEat an already-logged-in home; that is not repeated, and a real login stays owed (below).Still open (kept open on purpose)
test_repair_that_lands_on_a_known_bad_version_is_not_success). It cannot yet install an earlier good version. The known-bad list is empty today.auth_preflight, so neither startup nor the launcher stops. Onlyauth_preflight.py --live(one model call) catches it.PATHand proxy for an app started from the Dock, and re-running the check after the app quits mid-install. See the gap column of the acceptance matrix.Review round 5 (57cf9c2): the live witness, Susan's two open points, and a test that sent a real DM
Findings open after the reviews at
92ad0daf:1. REVIEW.md §15 live witness (review 5561694227). Partly done; the rest is owed and named below.
Done at the exact head
57cf9c2, on a real install, isolated from the live core:@openai/codex@0resolved to 0.163.0), realcodexbinary probed with--version/--help, and realcodex login status.src/agent/runtime-compat-gate.sh, off a TTY (stdin/dev/null), so--notify-owneris on.HOME,CODEX_HOME,SUTANDO_RUNTIMES_DIRand the workspace are in a scratch folder.PATHholds a brokencodexfirst, then onlynpm/node.SUTANDO_RUNTIME_SYSTEM_DIRS=so the host's own codex is not found.tasks/, the shape a bridge writes. No bridge ran, so no DM was delivered.The witness found a bug. The DM read "It changes only only <root>": the npm installer's text began with "only". Fixed in
57cf9c2; step 1 above is the fixed text. Pinned bytest_dm_names_what_the_npm_install_changes_in_one_sentence:Still owed for §15, and why I did not do it here:
start-cli.shcreates thesutando-coretmux session and loads launchd agents for the task notifier, next to the live core. Isolating that needs a throwaway macOS user or a test Mac.CODEX_HOMEis switched to an already-logged-in Codex home, which onlycodex login statusreads. A realcodex logininto the scratch home needs the owner's account.The steps for the owner's run are under "Not verified, and owed" below. They are unchanged, except that the desktop path now expects one DM, a real install, and then the login step when the account is logged out.
2. Susan, non-blocking: a registration test that runs the check list (review 5559989381).
test_registered_in_run_checksnow runsrun_all_checks()with everycheck_*stubbed. It asserts that thecore-cli-compatverdict is in the output; it no longer greps the source. Mutation: guard the call off withif False: checks.append(check_core_runtime_compat()).3. Susan, non-blocking: the remaining Codex login-check copies. (Fixed in round 6.) Filed as #5699, covering
src/startup-runtime.sh:73-80andsrc/verify-setup.sh:34-41. One finding is in the issue:startup-runtime.shexits 127 when the only working codex is the managed one offPATH, before the launcher's gate can find it. The matrix doc's "Not covered yet" links the issue.4. Found while testing: a test queued a real owner DM.
test_restart_with_unusable_cli_kills_nothingran the Claude launcher from the checkout. Off a TTY, the gate queues the owner DM into the checkout's resolved workspace. Run in a live checkout, the suite would have DMed the owner that the core could not start. Both Claude-launcher tests now run from a temp copy ofsrc/andscripts/(launcher_copy). The restart test also asserts exactly one DM in that copy's workspace.Results at 57cf9c2 (macOS, clean env:
env -i HOME PATH LANG USER TMPDIR)The new tests in this round pin existing behaviour and pass at
92ad0datoo; the one exception is the DM-wording test (the "only only" fix above). The fullcodex-core-launcherrun has 5 failures, all in the task-notifier tests:test_launches_codex_and_managed_task_notifiertest_managed_notifier_waits_for_each_result_before_next_tasktest_managed_notifier_does_not_recover_stale_running_from_PROSE_containing_the_placeholdertest_managed_notifier_does_not_treat_stale_running_no_affordance_as_idletest_unassigned_notifier_task_bypasses_slow_history_scanThe same suite fails the same way on the unmodified merge base
60bc40a4(origin/main), in the same clean env:Four are in both lists. The rest move between runs: they are timing bounds. For example,
test_unassigned_notifier_task_bypasses_slow_history_scanalone took 3.7 s and then 4.56 s, 4.66 s and 4.67 s at60bc40a4, and 4.64 s and 5.44 s at92ad0da, against a 4.0 s bound. CI is the arbiter; at92ad0dait was 34 SUCCESS and 1 SKIPPED.Acceptance matrix
Each row of the owner's acceptance matrix is marked one of three ways: existing (a test already in this PR), new (added in round 5) or slice 2 (the desktop installer, which this engine cannot cover). Test files:
rc=tests/runtime-compat.test.py,gate=tests/runtime-compat-gate.test.py,launcher=tests/codex-core-launcher.test.py.rctest_claude_minimum_supported_version_is_ok_and_one_below_is_too_old. existing:rctest_too_old(1.0.99),test_plan_too_old_claude_uses_its_own_updaterrctest_codex_minimum_supported_version_is_ok_and_one_below_is_too_old. existing:rctest_plan_too_old_codex_elsewhere,test_plan_too_old_npm_install_updates_that_prefixrctest_discovery_reads_version_and_capabilities(2.1.286),gatetest_healthy_host_keeps_the_same_commandrctest_codex_current_version_is_ok(0.160.1). existing:rctest_known_location_found_off_path. live witness: a real 0.163.0 from npm passed the proberctest_known_bad. new:rctest_known_bad_first_on_path_is_skipped_for_a_working_second,test_known_bad_alone_blocks_and_plans_a_consented_repair,test_repair_that_lands_on_a_known_bad_version_is_not_successrctest_codex_known_bad, plus the first test in the Claude cell, which runs both CLIsrctest_newer_untested_is_usable_with_warning(3.0.0),test_newer_untested_warns_and_continuesrctest_codex_newer_untested_is_usable_with_warning(1.0.0)health-checkreportswarnrctest_missing_when_nothing_installed,test_claude_native_install_without_node;gatetest_missing_cli_stops_with_fix_and_resume_commandlaunchertest_no_codex_anywhere_with_consent_installs_and_launches,test_working_managed_codex_off_path_is_found_and_launched;rctest_codex_without_npm_uses_homebrew_when_presentPATH(test_codex_only_managed_startup_passes_its_cli_checks,test_claude_only_managed_startup_passes_its_cli_checks,test_verify_setup_finds_the_managed_codex_and_its_login). slice 2: first install on a clean Mac from the desktop installerrctest_broken_*(7 cases),gatetest_restart_with_unusable_cli_kills_nothing. new:gatetest_consent_repair_launches_the_managed_claude(real Claude launcher)launchertest_off_tty_broken_cli_asks_the_owner_then_their_reply_repairs_and_launches. live witness: steps 1 to 3PATHrctest_multiple_path_installs_first_usable_wins,test_multiple_path_installs_keep_path_order_when_both_work,test_same_install_twice_on_path_is_one_candidate,test_known_location_used_when_path_install_is_broken;gatetest_fallback_install_is_launched_by_path. new:rctest_every_path_install_unusable_names_the_first_and_lists_allrctest_codex_multiple_path_installs_first_usable_wins;launchertest_two_codex_on_path_the_too_old_first_is_skipped_for_the_second(real Codex launcher, launches the second by full path)PATH, not the shell's. The engine falls back to known locations, but the desktop has to decide whether to pass the login shell'sPATHrctest_interrupted_update_keeps_old_version(npmkill -9),test_leftover_staging_is_reported_then_cleaned,test_broken_partial_binary_from_interrupted_update,test_broken_dangling_link_from_interrupted_update, the 9PublishFaultscases and the 5 killed-rollback casesrctest_blocked_download_retries_with_backoff_then_guides,test_blocked_download_recovers_on_retry,test_native_install_blocked_download,test_failed_repair_is_answered_with_the_next_step. new:rctest_blocked_download_reply_keeps_the_old_version_then_a_later_reply_installsrctest_codex_homebrew_blocked_download_retries_then_guidesHTTPS_PROXY. The desktop should pass it, or name it in the repair cardrctest_disk_full_checked_before_download,test_disk_full_during_download_is_not_retried. new:rctest_disk_full_checked_before_the_native_installer,test_disk_full_during_the_native_install_is_not_retried,test_disk_full_reply_is_answered_then_a_later_reply_installs_and_continuesENOSPCuses the same classifierauth_preflight, so this gate lets the launch go on. Onlyauth_preflight.py --live(one model call) catches itrctest_codex_logged_out_points_to_login,test_codex_login_dm_then_retry_now;gatetest_codex_logged_out_stops_then_login_lets_it_start. new:rctest_codex_expired_login_stops_with_the_login_step_then_retry_continues;launchertest_owner_install_reply_on_a_logged_out_codex_answers_with_the_login_step. live witness: step 3 (a real logged-out codex)gatetest_consent_runs_the_repair_and_the_launch_continues;rctest_reply_during_the_wait_continues_the_same_launch,test_reply_after_the_wait_window_is_handled_on_the_next_launch. new:gatetest_consent_repair_launches_the_managed_claude(real Claude launcher launches<runtimes>/bin/claude)launchertest_off_tty_broken_cli_asks_the_owner_then_their_reply_repairs_and_launches,test_owner_install_reply_on_a_logged_in_codex_answers_that_it_starts,test_no_codex_anywhere_with_consent_installs_and_launches. new: the disk-full and blocked-download reply journeys above. live witness: steps 1 to 5Review round 4 (4e0c1fc): idempotent rollback, and the Codex launcher's ordering
1. Rollback was not idempotent. If
_undowas killed after restoring the old folder but before deleting its journal, the next recovery saw "aside gone + placed" and deleted the restored original. Now:_write_journalwrites it atomically (temp file, thenfsync, thenos.replace, then a directoryfsync). It records thephase(started→rolling_back|committed) and the inode of the new install (new_ino, kept across the rename) and of the old folder (old_ino)._undowritesphase: rolling_backbefore it touches the link or any folder._undoremovesfinalonly when its inode is the new install's. It restores the backup only whenfinalis absent. A restored original is therefore never deleted, however many times recovery runs.Kill-then-double-recover tests. Each one kills the process at one rollback boundary, runs recovery (killed again at the same boundary where that applies), then runs it twice more. Each asserts the original exists, the link resolves, no
.swap,.replacedor.stagingfiles are left, and the journal is gone:test_owner_repro_undo_killed_before_the_journal_delete_then_two_recoveries(the owner's repro)test_undo_killed_after_the_phase_is_written_then_recovered_twicetest_undo_killed_after_the_link_is_restored_then_recovered_twicetest_undo_killed_after_the_new_folder_is_removed_then_recovered_twicetest_undo_killed_after_the_old_folder_is_restored_then_recovered_twiceBefore (7d7348c, new test file copied in) and after (4e0c1fc):
Not every row fails on 7d7348c for the same reason:
_rmtree,_write_journal).test_undo_killed_after_the_link_is_restored_then_recovered_twicepasses on both heads. A kill at that boundary leaves the backup in place, which the old code already restored correctly. It is kept as a guard row.2. Susan's blocker: Codex launcher ordering. The launcher's own
command -v codexwait loop exited 127 before the compat gate could run. Now:PATH, offers or runs the install (with consent), and waits for an install in progress (SUTANDO_CODEX_WAIT_TIMEOUTstill bounds that wait).Both new tests drive the real
src/agent/codex/cli/start-cli.sh:test_no_codex_anywhere_with_consent_installs_and_launches: no codex anywhere, npm present, consent given. The install runs and the core launches<runtimes>/bin/codex -C ....test_working_managed_codex_off_path_is_found_and_launched: a working managed codex not onPATHis found and launched.tests/start-cli-restart-inside-core-guard.test.sh(the Codex positive control, which expects the launcher to reach its codex check) still passes. Local coverage (runtime-compat.test.pyalone):runtime_compat.py99%,owner_reply.py96%.bash scripts/review-checks.sh: PASS.Review round 3 (7d7348c): the backup blocker and durable owner replies
Blocker: the backup was deleted even when publication failed. Publishing is now one journaled transaction (
_publishinsrc/runtime_compat.py):<root>/<cli>/.swap-<pid>-<ts>.json, is written atomically before anything moves.recover_swaps, under the repair lock, and at the start of everyensure) undoes it, or finishes it if it had committed. This touches only Sutando-owned files.The reviewer's repro, re-run (a rename failure after the old folder was moved aside):
Fault-injection tests (
PublishFaults). The setup is a working 2.1.0 in use and a broken 2.3.0 folder already present. Each test asserts that 2.1.0 and the link are intact, the 2.3.0 folder is restored, there is no dangling link, and no.staging,.replacedor.swapfiles are left:test_fault_after_staging: a guard; it passes on both heads because nothing has moved yet.test_fault_after_the_old_folder_is_moved_asidetest_fault_renaming_the_new_folder_into_placetest_fault_after_the_new_folder_is_in_placetest_fault_swapping_the_linktest_fault_in_the_live_probetest_live_probe_fails_restores_old_folder_and_linktest_process_killed_mid_swap_is_undone_by_the_next_run: the undo itself dies, then the next run recovers from the journal.test_committed_swap_left_by_a_killed_process_is_finishedRequired: #5670's discipline on the off-TTY reply path. #5670 (849cdca) is not on main. The shared step machinery therefore lives in
src/owner_reply.py(STEPS,finish), not in a third copy:applied(with its answer text),answeredandarchived, each persisted atomically (local_record.write_whole).answeredmark decides. When it is unset,local_task_protocol.find_result(live plus every archive layout, fixed nameresults/<task>.txt) decides whether a copy was already written. A delivered answer, even one archived by a bridge, is never re-sent.task_archive.archive_file) after the answer.finallywrites an answer for a reply whose action was interrupted.Tests (
DurableReplies):test_crash_after_the_reply_is_recorded_reruns_only_the_actiontest_crash_after_apply_does_not_repair_againtest_crash_after_the_answer_is_written_never_sends_it_twicetest_crash_after_answered_mark_archives_on_replaytest_answer_delivered_and_archived_by_the_bridge_is_not_resenttest_reply_after_the_wait_window_is_handled_on_the_next_launch: the 600 s wait elapses, then the reply arrives, then the next launch repairs and continues.test_unfinished_reply_survives_a_new_incidentBefore (846aad5, new test file copied in) and after (7d7348c)
The rename-boundary tests error at 846aad5 rather than fail: the
_renameseam did not exist there. The reviewer repro above shows the same boundary losing the old folder on that head.Local coverage (
runtime-compat.test.pyalone):runtime_compat.py99%,owner_reply.py96%.bash scripts/review-checks.sh: PASS.Review round 2 (846aad5): Mini's changes and the owner-side additions
Blocker: consent off a terminal. A core started by the desktop app or a supervisor has no one to answer a prompt, so:
--notify-owner, passed when stdin or stderr is not a TTY) sends the owner one DM per incident through the atomicresults/proactive-*.txtpublisher;<workspace>/state/runtime-compat-<cli>.json, and is retried until it is queued once;install compatible claude/install compatible codex, orretry nowafter a manual fix or login;The gate acts on an owner-tier reply whose whole message is one of those phrases:
task_archive.archive_file, after the result is written;This is the engine half the desktop repair card will call; the desktop side (slice 2) is a separate PR in the desktop repo.
Reusing #5670's helpers. #5670 is not merged, so the shared shape lives in one small new module,
src/owner_reply.py: owner-tier reply text, bare-phrase matching, the one-DM publisher, and answer-then-archive. #5670's_owner_action/_ensure_notice/_archive_taskcan move onto it after it lands; that is noted as a follow-up in the matrix. This PR adds no third copy.Should-fix 1, rollback. A failed post-swap verify now switches
bin/<cli>back to the previous target (or removes a link that did not exist before).Should-fix 2, same-version reinstall. A working existing
versions/<v>is reused untouched. A broken one is moved aside, replaced, then removed. The live target is never deleted.Should-fix 3, broken PATH binary. Known locations stand in when no PATH or managed install works, and never when a PATH install works.
Lock (now required).
assisted_installholdsfile_lock.locked_file(<root>/<cli>/.lock)around install and swap. A repair that waited and finds the link changed by the racing repair reuses that result.No npm / Node.
curl -fsSL https://claude.ai/install.sh | bash -s stable, which needs no Node. The documented installer has no target-folder option, so the consent prompt and DM say it changes$HOME/.local/bin/claudeand$HOME/.local/share/claude, plus one link in the Sutando-owned folder.brew install --cask codexwhen Homebrew is present (documented in the Codex README). Otherwise the user step is the documented standalone installer,curl -fsSL https://chatgpt.com/codex/install.sh | sh, or Homebrew, or Node and npm.Logged-out Claude. This is covered by the existing startup auth preflight (
src/auth-preflight-gate.sh/src/auth_preflight.py), which already stops a boot. The gate reuses that verdict and only advises.New tests: before (old head 2a54a72) and after (846aad5)
The old head was run with the new test files copied in, plus
src/owner_reply.pyso the import works:(
test_npm_missingandtest_repair_reports_failurealso fail at the old head; they changed with the renamedno-installerreason.test_working_path_install_is_never_overridden_by_a_known_locationandtest_reinstall_replaces_a_broken_copy_of_the_same_versionare guards that pass on both heads.)The end-to-end test drives the real Codex launcher, off a TTY, with a broken
codex:install compatible codexarrives.new-session ... <runtimes>/bin/codex -C ....tasks/archive.Local coverage on the new code (
runtime-compat.test.pyalone):runtime_compat.py99%,owner_reply.py95%.bash scripts/review-checks.sh: PASS.Default wait. It is now 600 s off a TTY too, but only when the CLI is unusable. Healthy hosts return at once. Test fixtures that stub an unusable CLI set
SUTANDO_RUNTIME_WAIT_S=0.What changed, and why
The requirement: when the Claude Code or Codex CLI is the wrong version, broken or logged out, the user gets a route back to a running core. No dead ends, no bare "failed".
src/runtime_compat.py(new, stdlib only) is the one owner of "is this core CLI usable, and what fixes it".PATHin order, then a Sutando-managed install, then known locations (only when no PATH or managed install works).--versionand--helpand given one class:ok,newer-untested,too-old,known-bad,incompatible(a launcher flag is missing),brokenormissing.claudeworks launches exactly the same command as before.codex login status. Claude: the existingsrc/auth_preflight.pyverdict (reused, not copied).src/agent/runtime-compat-gate.sh(new) is the shared shell glue. Both launchers call it before any path that replaces a core.Then run again: bash <repo>/src/agent/start-cli.sh.SUTANDO_SKIP_RUNTIME_COMPAT=1skips it. It fails open when Python or the module is absent.codex login statusblock is replaced by the gate (--enforce-auth), so that check has one owner.health-check.py: newcore-cli-compatcheck. It reports the same verdict, with the fix.python3 src/runtime_compat.py doctor [--json]is the user-facing doctor.docs/runtime-compat-repair-matrix.md(the repair matrix), plus a link fromruntime-dependencies.md, adocs/README.mdentry, acatalog.jsonentry and the regeneratedsrc-map.md.Files to read first
src/runtime_compat.py:probe,discover,plan,assisted_install,ensure.src/agent/runtime-compat-gate.sh.src/agent/claude/cli/start-cli.sh(check_core_cli) andsrc/agent/codex/cli/start-cli.sh.docs/runtime-compat-repair-matrix.md.Repair matrix
See
docs/runtime-compat-repair-matrix.md. Each row gives class, detection, user-executable fix, assisted fix, verification and test name. A second table covers disk full, blocked download, interrupted update, a new binary that will not run, and npm missing.Tests
The required list maps to these tests. All binaries are shell stubs on a temp
PATH.test_discovery_reads_version_and_capabilities,test_known_location_found_off_path,test_incompatible_when_flag_missingtest_missing_when_nothing_installed,test_known_bad,test_newer_untested_is_usable_with_warning,test_too_oldtest_multiple_path_installs_first_usable_wins,..._keep_path_order_when_both_work,test_known_location_never_replaces_a_path_install,test_fallback_install_is_launched_by_pathtest_interrupted_update_keeps_old_version(npmkill -9mid-install),test_leftover_staging_is_reported_then_cleaned,test_broken_partial_binary_from_interrupted_update,test_broken_dangling_link_from_interrupted_updatetest_blocked_download_retries_with_backoff_then_guides,test_blocked_download_recovers_on_retrytest_disk_full_checked_before_download(npm never runs),test_disk_full_during_download_is_not_retriedtest_codex_logged_out_stops_then_login_lets_it_start,test_auth_recovery_waits_for_login_then_continues,test_claude_auth_uses_auth_preflighttest_assisted_install_launches_before_reporting_success,test_link_that_will_not_launch_is_not_success,test_unusable_download_keeps_old_versiontest_consent_runs_the_repair_and_the_launch_continues,test_waits_for_the_user_fix_then_continues,test_restart_with_unusable_cli_kills_nothing(real Claude launcher)Before (origin/main 22e970b, new test files copied in)
The last line is the user-visible bug on main. A
--restartwhose CLI cannot run kills the live core first and only then fails.After (HEAD)
Real host, read-only (
doctor)This host has two
claudeinstalls onPATH. The first is a non-executable file, which the shell also skips.Existing launcher tests
claude/codexeither gotSUTANDO_SKIP_RUNTIME_COMPAT=1, or now copy the gate and use a stub that answers--version/--help. That coverscodex-core-launcher(its auth-failure test now runs through the module) andexternal-core-helpers.shutdown-sentinel-*(3 files),core-restart-heartbeat-and-cron-hold,start-cli-claude-*,start-cli-personal-claude-hook-wiring,start-cli-worker-bootstrap,worker-mode-is-gated-on-env,external-core-helpers.codex-core-launcherhas 4 failures locally. The same 4 (plus 1) fail on unmodified origin/main on this host, so they are host-specific (an inherited task-event-handler env, and a timing bound). CI is the arbiter.start-cli-model-pin,start-cli-proxy-routing,voice-preference-consumersandapp-watcher-notifier-livenessfail on this host on origin/main too.The second commit (2a54a72) fixes what the first CI run found:
SUTANDO_SKIP_RUNTIME_COMPAT=1(start-cli-model-pin,start-cli-restart-verify,start-cli-private-socket-foreign-core,start-cli-restart-inside-core-guard);core-cli-compatis registered insrc/recovery_issues.pyHEALTH_CAUSES;statetologin, which the state-paths lint read as a runtime-state path.CI at 2a54a72: 34 pass, 1 skipped. That includes
diff coverage >= 95% (python), all shell lanes andlicense/cla.bash scripts/review-checks.sh:PASS (hardcoded-paths + root-artifacts + prose-cap clean).Feature/documentation consistency
New canonical doc
docs/runtime-compat-repair-matrix.md, registered indocs/catalog.json.runtime-dependencies.mdlinks it, and itslast_verifiedis updated.Worst-case disruption, and mitigation
newer-untestedwarns and continues.SUTANDO_SKIP_RUNTIME_COMPAT=1is a one-line escape, and it is printed in the matrix.PATHbinary works, the launch command is unchanged (test_healthy_host_keeps_the_same_command). Known locations never displace aPATHinstall.login statusexit code). The "node not found" case is now reported by the probe asbroken, with a Node.js fix.Not verified, and owed
install compatible claude, and expect the core to come up.PATH=/usr/bin:/bin SUTANDO_RUNTIME_SYSTEM_DIRS= bash src/agent/start-cli.sh --runtime claudein a terminal. Expect the reason, the fix, the assisted command, and a 600 s wait.curl -fsSL https://claude.ai/install.sh | bash). Expect the first terminal to print "fixed; continuing the launch" and the core to come up.PATH(npm install -g --prefix /tmp/old @anthropic-ai/claude-code@1, then put/tmp/old/binfirst). Expecttoo-old. Answeryto the assisted fix and confirm the core starts on~/.local/share/sutando/runtimes/bin/claude.codex logout, then start with--runtime codex. Expect the stop withcodex login, then a resume after login.pgrep -x claudeshould still match; the live witness step 3 confirms it. The native and Homebrew installers are tested only with stubs.src/startup-runtime.sh, thesrc/startup.shprerequisite check andsrc/verify-setup.shnow use the gate.🤖 Generated with Claude Code