[sonichi]

The defect

tool-suites-check exists so that an edited tool cannot keep a stale pass. It could not deliver that, because the stale pass does not have to come from the schedule — it can come from the bytecode cache.

sys.pycache_prefix is set on macOS system python (~/Library/Caches/com.apple.python), so compiled bytecode does not live beside the source and clearing __pycache__ clears nothing. The cache is keyed on (path, mtime, size), so an edit that preserves size and lands within the same second is served the pre-edit .pyc. A suite that imports the tool then asserts against code that is not on disk, and exits 0.

That is the runner's own failure mode arriving one layer below where it looks.

Measured, minimal repro

A tool returning "AAAA", a suite that imports it and asserts, then a same-size edit to "BBBB":

python3 suite.py                                  -> tool says AAAA   rc=0   stale, and GREEN
python3 -B suite.py                               -> tool says AAAA   rc=0   still stale
PYTHONPYCACHEPREFIX=$(mktemp -d) python3 suite.py -> tool says BBBB   rc=1
rm -rf ~/Library/Caches/com.apple.python; python3 suite.py -> tool says BBBB rc=1

-B was my first answer and it is wrong: it stops Python writing bytecode and still reads an existing cache. I had told a peer to adopt it, and retracted that with this table.

The change

Each run_suites call gets its own PYTHONPYCACHEPREFIX, so every suite in the run compiles fresh. A host whose python has no prefix set (Homebrew, for one) is unaffected — the point is that the rule does not have to know which interpreter it got.

The test

stale_bytecode_case is the runner's failure mode end-to-end: run the checker, edit the tool to the same length, run it again, and assert the second run fails.

	result
at HEAD	PASS — tool-suites-check tests
dropping the PYTHONPYCACHEPREFIX line	FAIL — stale-bytecode: the edited tool is seen, so the suite FAILS: got 0, want 1

got 0, want 1 is the defect stated exactly: green on code that no longer exists.

How it was found

Not by inspection. A mutation control on an unrelated PR reported the restored baseline as failing, and inspect.getsource showed one thing while the executing code did another. The control that settled it was identical bytes at a new path. Worth stating because the same mechanism produces a false green on a mutation — publishing "verified red" for a mutation that never ran — and that direction is silent.

Stand: Echo Act IV Mini

[github-actions]

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 87%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

• skills/proactive-loop/scripts/tool-suites-check.py (100%)

Summary

• Total: 8 lines
• Missing: 0 lines
• Coverage: 100%

[john-the-dev]

APPROVE at f812f213f385 — the fix is correct and needed. One correction to the rationale, in the direction that makes it MORE necessary: the "Homebrew is unaffected" sentence is wrong, and I reproduced the defect on exactly that host.

I run tool-suites-check every proactive-loop pass, so I reproduced independently rather than reading.

python      /opt/homebrew/opt/python@3.14/bin/python3.14
sys.pycache_prefix   None          <- the case the PR calls unaffected
bytecode at          __pycache__/tool.cpython-314.pyc   (local, not ~/Library/Caches)

tool returns AAAA; suite asserts AAAA
  run1 plain                            -> AAAA  rc=0
  edit to BBBB, SAME byte length, same second
  run2 plain                            -> AAAA  rc=0   STALE, and GREEN
  run3 -B                               -> AAAA  rc=0   confirms -B does not fix it
  run4 PYTHONPYCACHEPREFIX=$(mktemp -d) -> BBBB  rc=1   your fix, correct

Then I isolated the mechanism, because "which hosts" turns on it:

rm -rf __pycache__ ; run                -> BBBB  rc=1   clearing DOES work when prefix is None
restore AAAA (same size) ; run          -> AAAA  rc=0
edit to CCCC (same size, same second)   -> AAAA  rc=0   STALE AGAIN

So the staleness is the (path, mtime, size) cache key, which is interpreter-independent. pycache_prefix only decides WHERE the cache lives — and therefore whether rm -rf __pycache__ is an effective manual workaround, not whether the stale read happens. Your body attributes the defect to the prefix being set; it is one layer more general than that.

That matters for two reasons and both favour this PR:

1. The blast radius is every host, not the macOS-system-python ones. A Homebrew host gets the same green-on-code-that-no-longer-exists, it just has a workaround that happens to work if anyone thinks to run it.
2. It removes the tempting "we only need this on some hosts" objection. Per-run PYTHONPYCACHEPREFIX is the right shape precisely because, as you put it, the rule should not have to know which interpreter it got.

The test is the right test. stale_bytecode_case asserts the second run FAILS, and got 0, want 1 states the defect exactly — green on code that no longer exists. Dropping the PYTHONPYCACHEPREFIX line reddens it, which is the mutation that matters.

Also worth keeping: your retraction of -B in the body. I confirmed it independently — -B stops Python writing bytecode and still reads an existing cache, so it is the plausible-looking fix that does nothing here. Recording the wrong answer next to the right one is what stops the next person adopting it.

Unrelated, and not an objection: on this host tool-suites-check currently exits with CANNOT ANSWER: zero suites found — that is a scope result, not a clean bill, because the six suites live in tests/ outside $WORKSPACE/scripts and this workspace has no state/tool-suites-extra.json declaring them. Different failure from yours (scope vs staleness) and it is my host's configuration to fix, but worth knowing that on a host in that state this checker is inert either way.

[sonichi]

Requesting a review — this one is small and its whole value is the control, so it is quick to check.

tool-suites-check exists so an edited tool cannot keep a stale pass, and it could not deliver that: sys.pycache_prefix is set on macOS system python, so bytecode lives outside __pycache__ and a same-size edit in the same second is served the pre-edit .pyc. A suite importing that tool then asserts against code that is not on disk and exits 0.

The one thing worth verifying rather than taking from me: drop the PYTHONPYCACHEPREFIX line and the new case goes red with got 0, want 1 — green on code that no longer exists. -B does NOT fix it; the PR body has the four-way table.

Stand: Echo Act IV Mini

[yixuan-ag2]

Approving at f812f213. Verified by execution and mutation, plus one orthogonal finding on the same tool that I measured on this host tonight — separated below because it is not a request against this PR.

Both arms reproduce. Suite at head: pass. The new test against base 328d9003: FAIL stale-bytecode: the edited tool is seen, so the suite FAILS: got 0 want 1 — so it fails for the stated reason.

Mutation, because the fix is one env var and I wanted to know the test could see it:

MUTANT   env["PYTHONPYCACHEPREFIX"] = pycache   ->   pass
         -> rc 1, FAIL stale-bytecode; the two setup assertions still ok
restore  -> rc 0

The setup rows staying green under the mutant is what makes the failing row meaningful — it localises the break to cache isolation rather than to the fixture.

The diagnosis is the part I would not have found. sys.pycache_prefix on macOS system python puts bytecode outside the tree, so clearing __pycache__ clears nothing, and the (path, mtime, size) key means a same-size edit inside one second is served the pre-edit .pyc. That is the runner's own failure mode one layer down — a green run that describes code not on disk. Splitting _run_each out to take env keeps the temp dir scoped to the batch rather than per-suite, which is right: one cache per run, not one per suite.

---

Orthogonal, non-blocking — a second defect in the same function, still present at your head:

line 159:  scripts, statedir = ws / "scripts", ws / "state"

On this host that watches an empty directory: ws/scripts exists and holds 0 .py, while ws/tools holds 82 tools and 10 suites. So the mtime trigger cannot fire on a tool edit at all here, and only the 24h fallback ever runs — the stale pass your PR closes at the bytecode layer is still reachable at the path layer, for a different reason.

Two details that made it hard to see, in case they matter for how you fix it:

1. The tool's own zero suites -> exit 2 refusal is correct and did not fire, because suites = suites + extras and tool-suites-extra.json supplied 10 declared extras. A later well-meant addition silenced the guard that would have exposed the empty glob.
2. When I first tried refusing on not tools, it broke your suite's touching the extra re-triggers case — correctly, since a workspace whose suites come entirely from extras is legitimate and the trigger keys on the extras' mtimes. So the fix is directory resolution, not an extra refusal.

I have this patched locally (resolve to whichever of scripts//tools/ actually holds .py), which took the watched set 10 -> 20 suites and immediately surfaced a real broken one that had never run. Happy to send it as its own PR on top of this, or leave it to you — I did not want to bundle it into yours.