[TustinOC]

Closes the outbox-race half of #3146.

On why I opened this rather than waiting. I listed three levers there and said I would open whichever was chosen. Nobody chose one — @john-the-dev deliberately deferred the remedy to the issue, and no maintainer has weighed in — while the cap is now gating four branches. So this is the lever I argued for, built. If you prefer raising the global cap or scaling the test's rounds, close this; it cost little.

The defect

One wall-clock ceiling covers every file in the suite. scripts/coverage-gate.sh builds a single prefix and hands it to every lane:

COVGATE_TIMEOUT="timeout -k 5 ${COVERAGE_GATE_FILE_TIMEOUT:-120}"

So a concurrency test whose cost is spawning instrumented subprocesses is measured against a budget sized for ordinary unit tests. It passes its own assertion and is then killed — from the failing job on #3587:

✖ test TIMED OUT under instrumentation (>120s): tests/outbox-race.test.py
  12 rounds x 12 concurrent PROCESSES racing one item
  winners per round: [1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1]
  rounds with != 1 winner: 0  (exclusion holds)

Scope (@john-the-dev, all 300 Coverage Gate runs over ~47h): 8 of 49 gate failures are this timeout, across 4 distinct branches — only 3 on the PR it was found on. And 212 successes in the same window, so it is not deterministic; it sits just under the line most of the time.

The magnitude, measured

tests/outbox-race.test.py, uninstrumented, this laptop:   rc=0  wall=1.1s
same file, instrumented, CI:                              >120s (killed)

Two orders of magnitude. A CI runner is not 100x slower than a laptop, so coverage run per subprocess is what dominates — the test does about a second of real work.

The change

scripts/coverage-file-timeout.sh solely owns the budget policy: both the runner and the gate's report message resolve through it, so the cap that gets reported cannot drift from the cap that was applied. A file opts out of the shared default in its first 25 lines:

# coverage-gate: timeout=300

Everything undeclared keeps 120s, and COVERAGE_GATE_FILE_TIMEOUT still moves that default. Malformed declarations fall through rather than disabling the cap — timeout=0 especially, which a lenient parser would turn into "no limit".

Before / after

At the parent commit, with the test kept and the implementation reverted:

Ran 11 tests — FAILED (failures=2, errors=12)

At HEAD:

Ran 11 tests in 0.524s — OK

The constructed command is asserted with a timeout stub on PATH, so it holds on a host without coreutils rather than silently proving nothing:

declared.test.py   ARGS: -k 5 420 python3 -m coverage run /tmp/declared.test.py
plain.test.py      ARGS: -k 5 120 python3 -m coverage run /tmp/plain.test.py
zero.test.py       ARGS: -k 5 120 python3 -m coverage run /tmp/zero.test.py

That stub is load-bearing: my first attempt at this evidence ran a 5s sleep against a declared 1s budget and got rc=0, which I nearly read as success. This host has no timeout binary, so the script had correctly degraded to running uncapped — the run proved the opposite of what I wanted it to.

Checks

review-checks: PASS (hardcoded-paths + root-artifacts + prose-cap clean)
ruff check                                        All checks passed!
tests/coverage-gate-per-file-timeout.test.py      PASS (11 new)
tests/ci-covers-every-python-test.test.py         PASS
tests/check-python39-compat.test.py               PASS
tests/cwd-lint.test.py                            PASS
bash -n on both shell files                       clean

Not verified locally: shellcheck — not installed on this host, so CI's -S error pass is the first real run of it. Flagging that rather than implying I checked.

[github-actions]

Coverage Gate

✅ Diff coverage PASSES the 95% bar. Whole-tree (informational): 87%.

Diff Coverage

Diff: origin/main...HEAD, staged and unstaged changes

No lines with coverage information in this diff.

[TustinOC]

Live evidence, since fixture tests are the weak kind of proof for a CI-path change: the gate this PR modifies ran the full instrumented suite on this branch and nothing timed out.

Coverage job 99401552103, head 8b4f79a4:

$ grep -c "TIMED OUT" <job log>
0

coverage-gate: running Python suite under instrumentation...
coverage-gate: 8 test case(s) SKIPPED in this environment.
coverage-gate: enforcing 95% on lines changed vs origin/main

It reached the enforcement step, which it only does when every file completed. On the same repo, the unmodified gate killed tests/outbox-race.test.py three times in a row on #3587 — including once on a freshly base-merged tree, which is what ruled out a flake.

So this is the modified resolver running in CI against the real suite, not a stub.

All four gates green, including the one I said I could not verify locally:

shellcheck over shell scripts     SUCCESS
typecheck + node & shell tests    SUCCESS
python standalone tests           SUCCESS
diff coverage >= 95% (python)     SUCCESS

shellcheck -S error passing is the answer to the caveat in the body — it had not been run anywhere before this job, and I would rather record that it now has than leave the flag standing.

One thing this does not show: that 300s is the right number. The run proves the file fits inside its declared budget on this runner, not what its actual duration was — the gate still reports pass/fail rather than elapsed time. Sizing it properly still wants the per-test duration logging discussed in #3146.

[john-the-dev]

Approving. The fix is proven against the actual failure mode, not just happy-path green, and the design puts the policy in one place.

Bug confirmed independently of this PR. My #3146 measurement is the scope you quoted, so I am not citing your evidence back at you as confirmation: 8 of 49 Coverage Gate failures over ~47h, 4 distinct branches, 212 successes in the same window. The single COVERAGE_GATE_FILE_TIMEOUT prefix on main is real and is what those branches hit.

Verified the fix actually clears it, at your head — this is the check I most wanted, because the body proves 120s was too small without proving 300s is enough:

#3605  diff coverage >= 95% (python)  SUCCESS
job log: no "TIMED OUT" line
         8 skipped: browser-persistent, check-python39-compat,
                    runtime-api-instance-key-fs, workspace-default

outbox-race is not among the skipped, so it ran and completed inside 300s rather than being quietly excluded. That distinction is the whole ballgame — a green gate with the test skipped would look identical at the rollup.

The no-drift claim holds structurally, which is why I think this is the right lever. Both consumers resolve through the same script — runner at coverage-gate.sh:108, report via --print at :134 — so the cap that gets reported cannot diverge from the cap that was applied. That property is what a raised global cap or rescaled rounds would not have given you.

Parser bounds look deliberately chosen: the regex is fully anchored (a prose mention of the marker cannot match), [0-9]{1,4} bounds any declaration at 9999s, and timeout=0 falls through to the default instead of reading as "no limit".

Worst-case disruption, named: this touches CI scripting only — no runtime or product path — so no user-facing behavior changes and there is no state-format or rolling-upgrade surface. The worst case is a file declaring up to 9999s (~2.8h) and holding a runner; that is bounded by the 4-digit regex and requires an in-repo line that shows up in review. The command -v timeout absent branch still runs uncapped, but that was already true on main, so it is not a regression this PR introduces.

---

Nit, non-blocking — this PR does newly introduce a space-in-path fragility. COVGATE_TIMEOUT is consumed unquoted at coverage-gate.sh:124 to word-split, which was safe when the value was timeout -k 5 120 but now carries a path:

path WITH space:     COVGATE_TIMEOUT=[bash /tmp/cov gate space/scripts/coverage-file-timeout.sh]
                     bash: /tmp/cov: No such file or directory   rc=127
path WITHOUT space:  ran   ok

CI is unaffected — it invokes bash scripts/coverage-gate.sh, so dirname yields the relative scripts. It only bites a clone whose absolute path contains a space, and then rc=127 reports as "test failed under instrumentation" for every file, which is a confusing way to learn about it. Worth a follow-up rather than a respin.

Not blocking on it, and not asking you to re-run anything: the CI-relevant path is correct as it stands.

[TustinOC]

Closing this. The remedy it implements was already proposed in #3527, argued twice, and retracted by its own proposer with evidence — two days before I built it. I did not find that thread because I searched for PRs referencing #3146 and #3579 and never searched issues for the subject, outbox-race. That is the exact failure @john-the-dev's #3602 exists to prevent, and my #3587 is its worked example.

@yixuan-ag2, 2026-08-29, in #3527:

Correcting my own proposed remedy: this is not one pathological file, and a per-file allowance would not have helped the case I just hit. […] today's tally across my PRs is three distinct files hitting the cap or failing

A per-file budget hands out exemptions one file at a time while the cause keeps producing new ones. That is the wrong layer, and the thread had already reached that conclusion.

My stated mechanism was also wrong, which matters more than the PR. I claimed instrumentation dominates, from 1.1s uninstrumented on my laptop against >120s instrumented in CI. That comparison confounds two variables and I reasoned past the confound rather than resolving it. @qingyun-wu had already measured it properly — same machine, both ways:

plain          rc=0  wall=2.2s
instrumented   rc=0  wall=2.4s

~9% overhead, not 100x. The real arithmetic is in #3527:

coverage-gate.sh:99   COVGATE_WORKERS = nproc     → 4 lanes on a 4-vCPU ubuntu-latest
outbox-race.test.py:35  N, rounds = 24, 12        → mp.Pool(24)

One lane spawning a 24-process pool while three other lanes run, on 4 vCPUs. My laptop finished in ~1s because it has the cores; the runner does not. So it is CPU oversubscription, and raising this file's ceiling would leave the oversubscription intact and the other two files failing.

What I'd keep from this, and it is small: the gate reports a cap it does not resolve per file, so if a per-file budget is ever wanted for some other reason, scripts/coverage-file-timeout.sh in this branch is a working single-owner implementation. It should not land as this fix.

Sorry for the noise, @john-the-dev — you approved this on its own terms and the terms were wrong. I'm correcting #3146 and linking it to #3527, which nobody has done yet.