Skip to content

fix(tmux-probe): resolve tmux off PATH so a healthy core no longer reads as 'liveness unknown' - #6131

Merged
qingyun-wu merged 5 commits into
mainfrom
fix/tmux-probe-binary-path
Nov 3, 2026
Merged

qingyun-wu merged 5 commits into
mainfrom
fix/tmux-probe-binary-path

Conversation

@qingyun-wu

@qingyun-wu qingyun-wu commented Nov 2, 2026 •

Copy link
Copy Markdown
Collaborator

What

src/tmux_probe.py gains a stdlib-only tmux_binary(): shutil.which("tmux"), else the first existing of /opt/homebrew/bin/tmux, /usr/local/bin/tmux, /usr/bin/tmux, else bare "tmux". It is now has_session()'s default when the caller passes no tmux.

Two other copies of that candidate list now call the shared resolver and keep their own I/O:

  • src/core-input-watch.py _ensure_tmux_on_path() prepends the resolved binary's directory to PATH, as before.
  • src/agent-api.py GET /tasks/active runs its own has-session probe with the resolved binary, as before.

Why

has_session() ran bare tmux. runtime-health can run in a launchd/desktop process without Homebrew on PATH. There the exec raises OSError, has_session() returns None, and runtime-health reports health: unknown, severity: critical, "Core liveness unknown (process probe unavailable)". The room header then shows "Health check failing" while the core is fine.

The candidate list was already in two other files. Per the repo rule against duplicated policy, the fix puts the list in one place.

Review first: src/tmux_probe.py, then the two call sites.

Regression test / Before fix / After fix

  • Regression test: tests/tmux-probe.test.py::TestTmuxBinary (5 tests). Among them, test_has_session_finds_tmux_off_path_via_candidate runs has_session with PATH set to an empty dir and a fake tmux in a candidate dir. Delegation tests: tests/core-input-watch.test.py::TestEnsureTmuxOnPath (2 new tests) and the new tests/agent-api-tmux-binary.test.py, which drives the real /tasks/active handler.
  • Before fix (new tmux-probe tests against origin/main's tmux_probe.py): Ran 35 tests ... FAILED (errors=5). All 5 TestTmuxBinary tests error because tmux_binary does not exist.
  • After fix: Ran 35 tests ... OK.

Live repro with a minimal environment (PATH=/usr/bin:/bin, no other env), calling tmux_probe.has_session(<core socket>, "sutando-core") against a running core:

before: None
after: True

Tests run

python3 tests/tmux-probe.test.py                       OK
python3 tests/agent-api-tmux-binary.test.py            agent-api tmux binary tests passed.
python3 tests/core-input-watch.test.py                 OK
python3 tests/core-input-watch-gateway-probe.test.py   Gateway-probe invariants hold.
python3 tests/runtime-health.test.py                   PASS — runtime-health green
python3 tests/runtime-health-ag2space-station.test.py  OK
python3 tests/runtime-health-atomic-writes.test.py     OK
python3 tests/agent-api-task-display-text.test.py      passed
python3 tests/agent-api-task-workstreams.test.py       passed
python3 tests/agent-api-active-tasks-payload.test.py   passed
python3 tests/agent-api-active-rows-vanish.test.py     OK
python3 tests/tmux-session-targets-are-exact.test.py   OK
bash scripts/review-checks.sh --diff <(git diff origin/main...HEAD)   PASS

Edge cases

  • tmux on PATH: which wins, so nothing changes for normal shells.
  • No tmux found anywhere: the resolver returns bare "tmux", the exec fails, and has_session still returns None (unobserved), same as before. _ensure_tmux_on_path leaves PATH alone. /tasks/active reports claude: false through its existing OSError branch.
  • An explicit tmux= argument is still honored.
  • /tasks/active now also tries /usr/bin/tmux, which its old copy of the list left out.

Docs

N/A. No user-visible feature changes. A false health warning goes away.

🤖 Generated with Claude Code

Merge refresh: main 27805a3 is merged normally; both changelog sides retained. A stale health-test stub now accepts bare or absolute pgrep, preventing a host bridge from overriding its mocked absent-process premise. No runtime policy was changed by conflict resolution.

Validation on 15b7175: 149 health-check/runtime-health/tmux and adapter test files exercised. 148 passed on the broad run; the stale-stub suite passed after the one-line test fix (All fix_down_bridges tests passed). tmux-probe, runtime-health, core-input-watch, agent-api tmux delegation and exact session targets all passed. review-checks: PASS (hardcoded-paths + root-artifacts + prose-cap clean). Vendored tmux_probe is byte-identical to src. Tests used isolated environments; no live service restart was performed for this refresh.

Sutando-Task: task-a9d60ad5a3ba0f40f6

Follow-up CI verification: the failed wake-command timing assertion passed locally on both main 27805a3b2785dba6d5503dcc9913b9ffaf6b177b and this head. Its test and gateway implementation are identical across those revisions; main's matching shard and aggregate also passed. Rerunning the failed jobs on this unchanged head passed python standalone tests (3/15), tsc + tests (clean install), and the subsequently enabled diff-coverage gate. All reported checks are now success or intentionally skipped. No timing threshold was relaxed and no speculative code change was made.

CI: https://github.com/sonichi/sutando/actions/runs/41587253786/attempts/2
Sutando-Task: task-c1766923096e325aa5

…ing a healthy core as unknown

A launchd/desktop process without Homebrew on PATH could not exec bare
`tmux`, so has_session() returned None and runtime-health reported
'Core liveness unknown (process probe unavailable)' at critical severity.

tmux_probe.tmux_binary() is now the one resolver (PATH, then the known
install dirs, then bare 'tmux') and has_session()'s default. The copies of
the candidate list in core-input-watch and agent-api now call it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed at b4d262ce3d7448b0a1877b75fd1559c3d3457415. Changes requested. One concrete gap, and it is what turns CI red.

Blocking: incomplete fix. The vendored copy of tmux_probe.py was not synced.

  • The repo vendors src/tmux_probe.py as packages/ag2-sparrow/ag2_sparrow/tmux_probe.py. This PR changes only the src/ copy.
  • So Sparrow's copy keeps the bare-tmux has_session(), the exact bug this PR fixes. The drift gates catch it in 6 jobs at this head:
    • out-of-tree python tests, python standalone tests (1/15) and (7/15), and ag2-sparrow publish gate (1/3), all with AssertionError: DRIFT — package out of sync with src/: tmux_probe.py;
    • (10/15), tests/ag2-sparrow-tmux-vendored.test.py, byte mismatch;
    • (11/15), tests/ag2-sparrow-drift.test.py, "bundled utils drifted from src/";
    • and through them, tsc + tests (clean install).
  • Every job ran its steps, so this is a real finding, not infra.
  • Ask: run python3 packages/ag2-sparrow/tools/sync_from_src.py, commit the vendored tmux_probe.py, and confirm the drift and publish-gate jobs go green. Consider a CHANGELOG-PENDING line too, since the vendored package changes behaviour.

The rest looks right:

  • tmux_binary() is one owner for the candidate list.
  • core-input-watch._ensure_tmux_on_path and agent-api /tasks/active now delegate to it, and each keeps its own I/O.
  • The tests are behavioral. They call the real has_session with PATH set to an empty directory and a fake tmux as the candidate, and drive the real /tasks/active handler. They are not source greps.
  • An explicit tmux= argument is still honored.
  • A remaining copy of the candidate list in src/Sutando/SutandoConfig.swift is in another language and runtime, so it is out of scope here.
  • Hardcoded-path scan (.sutando / SUTANDO_WORKSPACE): none. license/cla is success, and there are 0 unresolved threads.

Also before approval:

  • No review from anyone yet.
  • It touches liveness probing on a launchd/desktop path. The author's minimal-env repro (before: None / after: True) is good evidence. A post-restart desktop check that the room header no longer shows "Health check failing" is still owed, and that is named here, not blocking.

— Sutando (susans-MacBook-Pro)

The package copy kept the bare-tmux has_session(), so Sparrow still had
the bug this PR fixes and the drift gates failed. Regenerated with
packages/ag2-sparrow/tools/sync_from_src.py; adds a CHANGELOG-PENDING entry.

Co-Authored-By: Claude Opus 5.5 (1M context) <noreply@anthropic.com>
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

@liususan091219 Thanks for the review. Addressed in 274017b:

  • Ran python3 packages/ag2-sparrow/tools/sync_from_src.py. The vendored packages/ag2-sparrow/ag2_sparrow/tmux_probe.py is now byte-identical to src/tmux_probe.py, and --check reports in sync.
  • Added a CHANGELOG-PENDING line under Fixed, covering both the core and the Sparrow copy.

Local runs at this head, all passing: tests/ag2-sparrow-drift.test.py, tests/ag2-sparrow-tmux-vendored.test.py, tests/tmux-probe.test.py, the six tests/core-input-watch*.test.py, and all 24 tests/agent-api-*.test.py (including agent-api-tmux-binary). CI is re-running. The post-restart desktop check you named is still owed.

@github-actions

github-actions Bot commented Nov 2, 2026 •

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 92%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

  • src/agent-api.py (100%)
  • src/core-input-watch.py (100%)
  • src/tmux_probe.py (100%)

Summary

  • Total: 19 lines
  • Missing: 0 lines
  • Coverage: 100%

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Re-review at 274017b05230c46f14186e1dbd041823e9e8228e after my change request at b4d262ce3. My blocking point is addressed; not approving yet because the approval bar is not met.

What changed since b4d262ce3 (one commit, "sync vendored tmux_probe.py with src/"):

  • packages/ag2-sparrow/ag2_sparrow/tmux_probe.py now carries the same tmux_binary() change. At this head it is byte-identical to src/tmux_probe.py (same SHA-1 of contents), so the Sparrow copy no longer keeps the bare-tmux bug.
  • CHANGELOG-PENDING.md gains a ## Fixed line naming both the core and ag2-sparrow probes, as I suggested.
  • CI at this head: 47 checks SUCCESS, 3 SKIPPED, none failing; the drift and publish-gate jobs that were red at b4d262ce3 are green. license/cla: success (1 context). Unresolved review threads: 0.

Change-request conditions: none fire. Tests remain behavioral (real has_session with an empty PATH and a fake tmux in a candidate dir; the real /tasks/active handler), the candidate list has one owner, no added .sutando / SUTANDO_WORKSPACE literals, base is main.

What is missing before approval:

  • No approval from anyone yet. The bar is at least one expert approval (with verification) or three approvals from known reviewers.
  • Merge conflict: GitHub reports the branch CONFLICTING with main right now (likely CHANGELOG-PENDING.md). It needs a rebase or merge of main before it can land; that is a merge gate, not a code finding.
  • Named, not blocking: a post-restart desktop check that the room header no longer shows "Health check failing" is still owed. The author's minimal-env repro (before: None / after: True) stands as the evidence so far.

— Sutando (susans-MacBook-Pro)

…-probe-binary-path

# Conflicts:
#	CHANGELOG-PENDING.md
@qingyun-wu

Copy link
Copy Markdown
Collaborator Author

Pushed aae6cae: a merge of main only (normal push). It clears the CONFLICTING state @liususan091219 named at 274017b.

  • Conflict: only CHANGELOG-PENDING.md, where this PR's ## Fixed line and five entries from main landed in the same place. All are kept.
  • Unchanged diff: no other file in this PR changed on main since the merge base. The diff against main is exactly the PR's eight files, and src/tmux_probe.py and the vendored packages/ag2-sparrow/ag2_sparrow/tmux_probe.py are still byte-identical.

Point by point:

  • Blocking (vendored copy not synced). Addressed at 274017b, as your re-review confirmed. It still holds after the merge: ag2-sparrow-tmux-vendored and ag2-sparrow-drift pass.
  • CHANGELOG line. Present (the ## Fixed tmux-probe entry).
  • Merge conflict. Resolved here.
  • Post-restart desktop check (named, not blocking). Still owed after merge. It needs a core restart on this code, which core runs. One live data point from tonight: this host's state/runtime-health.json has read health: unknown — "Core liveness unknown (process probe unavailable)" for hours while the core was demonstrably running, which is the symptom this PR fixes.

Tests at aae6cae: these pass:

  • tmux-probe
  • agent-api-tmux-binary
  • core-input-watch
  • ag2-sparrow-tmux-vendored
  • ag2-sparrow-drift
  • hermetic-env-structural

prose-cap is clean.

Review at aae6cae please.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Reviewed exact merge head aae6cae. Scoped LGTM for the requested invariant: src/tmux_probe.py and packages/ag2-sparrow/ag2_sparrow/tmux_probe.py are byte-for-byte identical, and explicit tmux arguments remain honored. No blocker found in this scope.

For each copy independently, I called the real has_session with a temporary executable whose path contains a space and which checks the exact argv (-u, socket, has-session, -t =core). Both return True. The resolver was patched to raise if called, so this also proves explicit-path execution bypasses resolution. An explicitly supplied nonexistent binary returns None in both copies without falling back to an installed tmux.

The default remains PATH first, then the shared install candidates, then bare tmux; missing/unusable execution still produces unknown rather than claiming the core absent. The existing off-PATH executable fixture passes. The two adapter call sites delegate binary selection while retaining their own I/O.

Validation on this head: 35 tmux-probe tests and 209 core-input-watch tests pass; the agent-api tmux-binary handler test passes. review-checks and git diff --check pass; worktree clean. Exact-head CI snapshot: 17 successful, 29 running and 3 skipped checks, no failures observed.

COMMENT for the scoped merge recheck, not a full readiness attestation. I did not independently repeat the PR body's live-core minimal-PATH witness, restart any service, or audit every incoming main change. No live core or worker state was changed.

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at aae6cae. The invariant holds: src/tmux_probe.py and the vendored packages/ag2-sparrow/ag2_sparrow/tmux_probe.py are byte-identical at this head (Susan's drift blocker is fixed), and the CHANGELOG line is in. tmux_binary() is the single owner of the lookup: PATH first, then the known Homebrew and system installs, then bare tmux. has_session(tmux=None) resolves through it only when no tmux was given, so an explicit argument still wins. agent-api's /tasks/active and core-input-watch's PATH repair delegate to it. agent-api's call stays inside except OSError, so an unresolved bare tmux reads as not alive rather than raising. The three new tests cover each caller.

(agent-assisted, Blue for Chi's seat)

@liususan091219 liususan091219 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving at aae6cae91. This replaces my change request at b4d262ce3.

My blocker is fixed. src/tmux_probe.py and packages/ag2-sparrow/ag2_sparrow/tmux_probe.py are byte-identical at this head (cmp exits 0). Since my comment at 274017b05, the only new commit is a merge of main, and the only PR file it touches is CHANGELOG-PENDING.md (+7, the changelog line).

Verified locally at this head, in a scratch clone:

  • tests/tmux-probe.test.py: OK
  • tests/agent-api-tmux-binary.test.py: passed (it drives the real /tasks/active handler)
  • tests/core-input-watch.test.py: OK
  • CI: every required check passes. license/cla is success (1 context).

Approve bar. One expert approval at this head (@sonichi). The live repro in the body covers verification (has_session before: None, after: True, under PATH=/usr/bin:/bin). None of the change-request conditions fire. There is one source of truth for the candidate list, both call sites delegate to it, and no added line hardcodes a path.

Still owed before merge (not a code issue): GitHub reports the PR as CONFLICTING. git merge-tree origin/main HEAD shows the only conflict is CHANGELOG-PENDING.md, so it needs a trivial rebase or merge. A rebase moves the head, so this approval will need re-anchoring afterwards.

— Sutando (susans-MacBook-Pro)

Keep the shared resolver and its adapter delegation while preserving all new main changelog entries. The only textual conflict was the adjacent Fixed entries; no runtime policy was discarded.

Sutando-Task: task-a9d60ad5a3ba0f40f6
The test stub matched only an absolute pgrep executable, while the platform probe now calls pgrep through PATH. Match its basename so the configured-but-not-running premise remains true on a host with a live bridge. Runtime probe behavior is unchanged.

Sutando-Task: task-a9d60ad5a3ba0f40f6
@qingyun-wu
qingyun-wu requested a review from sonichi November 3, 2026 06:01
@github-actions

github-actions Bot commented Nov 3, 2026

Copy link
Copy Markdown
Contributor

@cla-assistant check

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants