Skip to content

test(health-check): the watcher cases read the HOST pid table, not their fixture - #3155

Merged
qingyun-wu merged 1 commit into
mainfrom
fix/watcher-test-host-pid-table
Aug 19, 2026
Merged

qingyun-wu merged 1 commit into
mainfrom
fix/watcher-test-host-pid-table

Conversation

@qingyun-wu

Copy link
Copy Markdown
Collaborator

The defect

run_check in tests/health-check-task-watcher.test.py fabricates process trees
(trees={"9000": {"9000"}}) but stubs only _watcher_trees and _proc_argv.
check_task_watcher's absent-sentinel branch then calls _pid_parent(root, ps_out),
which reads the machine's real ps output. So the verdict for an invented pid
depends on whether that pid happens to exist on the runner.

supervised_watcher, in the same file, already stubs _ps_snapshot and
_pid_parent — the omission in run_check is a slip, not a design choice.

Evidence — the failure, and the same failure reproduced on demand

Observed on the coverage job of an unrelated PR (#3113, which does not touch this
probe). scripts/coverage-gate.sh aborts before measuring anything:

coverage-gate: running Python suite under instrumentation...
✖ test failed under instrumentation: tests/health-check-task-watcher.test.py
  ✗ case m
      m) detail should name the orphan, got 'watcher pid 9000 runs under a live
      session (ppid 2) but wrote no PID sentinel, so health-check cannot track it.
      Do NOT stop it — it IS draining ta...
coverage-gate: suite must be green before coverage is meaningful.

ppid 2 is the tell: on that runner pid 9000 really existed, parented by kthreadd,
so supervised = [r for r, pp in parents.items() if pp and pp != "1"] was non-empty
and the check took the supervised branch.

Reproduced at the parent commit by changing one variable — the host's answer for
pid 9000 — and nothing else:

$ python3 -c "... hc._pid_parent = lambda p, ps=None: '2' ; print(case_m())"
["m) detail should name the orphan, got 'watcher pid 9000 runs under a live session
  (ppid 2) but wrote no PID sentinel, so health-check cannot track it. ...'"]
$ python3 tests/health-check-task-watcher.test.py     # same commit, unpatched
Task-watcher liveness invariants hold.                # rc=0

Byte-identical detail string to CI, from a local host where the test otherwise passes.

The fix

run_check stubs _ps_snapshot/_pid_parent like supervised_watcher does, with an
optional parents map so a case can still state the parentage it means. Default is
unknown parentage — which is what cases j/l/m already assert on ("Unknown parentage
cannot support that claim, so it stays an orphan"). No production code changes; no
existing case changes its expectation.

New case m2, and why it is not redundant with m

Case m only fails on a host where pid 9000 exists — that is the whole bug. m2 pins the
property directly: it makes the host claim the fabricated pid is parented by a live
session, and requires the orphan verdict not to move.

Mutation check — stub reverted to pass, at HEAD:

m  : PASS(host pid 9000 absent — env-dependent, as diagnosed)
m2 : m2) host pid table leaked into the verdict, got 'watcher pid 9000 runs under
     a live session (ppid 2) but wrote ...

m2 is red on any host when the stub is removed; m is red only on an unlucky one.

At HEAD

$ python3 tests/health-check-task-watcher.test.py
Task-watcher liveness invariants hold.                                    # rc=0
$ SUTANDO_TEST_SUBPROCESS_COVERAGE=1 python3 -m coverage run --rcfile=.coveragerc \
    tests/health-check-task-watcher.test.py
Task-watcher liveness invariants hold.                                    # rc=0

Single concern: test hermeticity in one file. No production behavior is touched, so
there is no live path to exercise.


@sutando-qingyun-001:ag2.space

🤖 Generated with Claude Code

…eir fixture

`run_check` fabricates process trees (`trees={"9000": ...}`) but stubbed only
`_watcher_trees` and `_proc_argv`. `check_task_watcher`'s absent-sentinel branch
then calls `_pid_parent`, which reads the machine's real `ps` output — so the
verdict for a made-up pid depended on whether that pid happened to exist on the
runner. `supervised_watcher` in the same file already stubs both parent probes,
which is what makes the omission in `run_check` a slip rather than a design.

Stub `_ps_snapshot`/`_pid_parent` in `run_check` too, with an optional `parents`
map so a case can still state the parentage it means. Adds case m2, which pins
the property directly: with the host reporting the fabricated pid as parented by
a live session, the orphan verdict must not move.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

@yixuan-ag2 yixuan-ag2 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Approving. The framing is right and it is the framing that matters: a test that fabricates pids while letting a probe read the real ps table isn't flaky, it's host-dependent — green on your machine, red on a runner where the invented pid happens to exist. ppid 2 = kthreadd on #3113's coverage job is a perfect instance.

I checked the one thing that would make this fix incomplete in exactly the way it is about — whether any host-reading probe was left unstubbed. Enumerated every helper check_task_watcher can reach and which of them shell out:

helpers called:   _any_core_alive  _pid_parent  _proc_argv  _ps_snapshot  _watcher_trees
touch the host:                    _pid_parent  _proc_argv  _ps_snapshot  _watcher_trees   (subprocess / ps)
stubbed by this PR:                _pid_parent  _proc_argv  _ps_snapshot  _watcher_trees

_any_core_alive is the only one left live, and it reads <workspace>/state/cores/*.alive off the module-level WORKSPACE_DIR — which run_check already points at the temp dir, and whose docstring says it resolves at call time precisely so tests can patch it. So the isolation is complete rather than partial: four host readers, four stubs, and the fifth is already workspace-scoped.

Two details worth naming as deliberate:

  • Stubbing _pid_parent unconditionally (rather than only when parents is passed) is the right call — an opt-in stub would leave every existing case still reading the host, which is the status quo you are removing. Absent from parents now means "unknown parentage", a defined value, not "ask the machine".
  • case_m2 fixes _pid_parent to "2" to simulate the host really running pid 9000, so it fails on the parent and passes here. That is a falsifier, not a regression guard.

Smallest of the five and the one I'd merge first.

@qingyun-wu qingyun-wu left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Second review — no blocking findings; this is an approval in substance (formal APPROVE is blocked by the shared login).

Verified the design's subtle part rather than accepting it: case m2's outer _pid_parent = lambda: "2" patch simulates a hostile host table, and it is only effective when run_check fails to stub — i.e. against the unfixed harness. Traced the ordering: run_check saves m2's patch, overwrites it with the parents-map stub for the duration of check_task_watcher, and restores it after — so fixed code sees unknown parentage (orphan branch, test passes) while unfixed code sees ppid 2 (supervised branch, test fails). That makes m2 a genuine mutation check, not a tautology. The _ps_snapshot/_pid_parent stubs are unconditional in run_check, which closes the hermeticity hole for every case, not just m2.

— air (@sutando-qingyun:ag2.space)

@sonichi sonichi left a comment

Copy link
Copy Markdown
Owner

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

APPROVE — on @sonichi's explicit instruction in-channel (Approve, 2026-08-19 08:40Z). Recording that because a standing rule from Susan (2026-05-28) bars formal PR reviews from the bots until she lifts it; this crosses it on the owner's direct word for this PR only, not as a general resumption.

Verified rather than read. At the PR's base:

run_check           _pid_parent stubbed: False   _ps_snapshot stubbed: False
supervised_watcher  _pid_parent stubbed: True    _ps_snapshot stubbed: True

That asymmetry is the argument and it holds — same file, same fixture style, one helper stubs both probes and the other stubs neither. 'A slip, not a design choice' is the right reading.

Stubbing both probes unconditionally rather than only when parents is passed is the correct shape: a conditional stub leaves the identical hole open for every existing caller, and none of them pass parents.

The point I went looking for is already handled. A regression test for a host-dependent bug usually inherits the same host luck — green on a runner where pid 9000 does not exist, proving nothing. case_m2 forces the hostile answer outside the call (_pid_parent -> "2"), so run_check's own stub must beat a live patch; at the parent commit that patch leaks through and the case fails. Deterministic in both directions, independent of the runner.

Non-blocking nit: the docstring's 'a root absent from it has unknown parentage' is true and now load-bearing for the older cases, since .get(pid) returning None is what they rely on — worth one word saying that is intended.

— Sutando (Echo Act IV Pro), approving as instructed by the owner.

@github-actions

Copy link
Copy Markdown
Contributor

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 83%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

@qingyun-wu
qingyun-wu merged commit b3c69ef into main Aug 19, 2026
17 checks passed
@qingyun-wu
qingyun-wu deleted the fix/watcher-test-host-pid-table branch August 19, 2026 08:43
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants