Closed
Description
Part of the epic "Epic: slice 0, foundations. The repo holds the line, its context is tested, and the platform facts are recorded".
Why. On "CI: a Lens's verdict and the secret scan's rules run from the base" (#145), a local python3 -m tools.mutants reported four mutants killed that CI's shards reported SURVIVED. The code they mutated had become dead; no test could kill them. Locally they were "killed" by an unrelated test failing under load (the Mac's load average was near 30 with several agents running): tools/mutants.py counts any failing test as a kill. So a flaky or timing-sensitive test failing for its own reasons hides a rule nothing checks — the exact thing the mutation check exists to catch.
Done when
- A mutant is killed only when a test fails for that mutant: the unbroken suite's failures, and any test that fails on a re-run of the unbroken code, do not count (for example: a kill must be reproduced, or must come from the tests the mutant names first, or a timing test is excluded from the mutation run).
- Timing-sensitive tests (a deadline in the hooks, a budget) are marked and kept out of the kill count, or made deterministic.
- A test holds that a mutant whose only failure is an unrelated test is reported SURVIVED.
Metadata
Metadata
Assignees
Labels
No labels
Activity
don-petry commentedon Oct 17, 2026
Working rulings (overturnable in review)
Solution Architect, delegated. On main at 3d4109c (#145). 404 mutants today, 16 with named tests.
One correction first. A timeout is not a kill today.
verdict()makes it anERROR, which fails the run. The false kills in #145 came from a failing test: each run stops at its first failure (-f), and any failure counts.1. What a kill is. A kill is a test that fails on the mutant, fails again when run alone on the mutant, and passes when run alone on the unbroken code.
attempt()reads the one failing test from the output. It reads both forms of the line:FAIL: test_x (tests.m.T)(3.9, 3.10) andFAIL: test_x (tests.m.T.test_x)(3.11 and later, which CI runs). It also reads a subtest suffix andsetUpClass, which names the class. A line it cannot read is anERROR.run(work, only=id)andrun(root, only=id).SURVIVED, and a confirmed failure iskilled. A second unconfirmed failure isFLAKY.Why: only re-running on both sides tells three failures apart: one this mutant causes, one already broken, and one caused by load. In #145 the re-run on the unbroken code alone would still have passed. The cost is two single-test runs a mutant, about 35 s a shard. The "tests first" field stays, only for speed.
2. Timing-sensitive tests. They are marked
@wall_clockintests/support.py, which skips them underMUTATINGas@pendingdoes. They still run inmake verifywith their margins as they are.test_hooks.py:Identity.test_a_check_that_runs_out_of_time_refuses,test_the_brace_bomb_is_answered_ask_within_the_deadline,test_a_brace_left_open_is_read_within_the_budgetandtest_a_wildcard_that_would_backtrack_is_read_within_the_budget.test_tools.py:test_a_run_that_hangs_is_ended_reported_and_fails_the_checkandtest_mutants_are_reported_in_their_order_however_their_runs_finish.test_a_parse_over_its_budget_asksandRulesChanged'sseconds=0case, since load only makes a spent budget more certain. The 60 s subprocess timeouts are not marked either; the confirmation covers them.Why: no mutant touches a hook's budget, so skipping these tests loses no kill. A test holds the mark.
3. Timeouts. A timeout stays an
ERROR, on any run, including a confirmation. It fails the run. It is never killed and neverFLAKY. Why: 600 s is 10 to 30 times a full run, so a timeout means a hang. A rule that only a hang shows needs a test that fails.4. Reporting.
FLAKY <mutant> (<test id> passed when run again on the mutant | failed on the unbroken code too; mark or mend that test, then run python3 -m tools.mutants again).FLAKYfails the run asERRORdoes, and the summary line does not change. Why: a mutant nobody showed is killed is a rule unchecked. Passing it would bring back #145 in a quieter form.5. Tests, mutants, record.
test_tools.py::Mutants. The issue's case:test_a_mutant_whose_only_failure_is_unrelated_survives, once withtold()and once against a realsuite_ofwhose test fails on its first run only. Also tests for:FLAKY;mutants.py, with the text split in two as the shard mutants are, and the new tests named as their fifth part):FLAKYcounted as killed;@wall_clocknot skipping underMUTATING.tools/AGENTS.mdrow "Only a failing test kills a mutant" to say the above. Add a lesson, "A test that failed under load killed a mutant", with these checks.docs/decisions/0027-a-mutant-is-killed-by-a-test-that-fails-on-it-twice-and-passes-without-it.md, new, replacing nothing. It records the alternatives not taken: named tests only, and no-f.Budgetinprotected.pyand the clock inidentity.pytake an injected clock, and add mutants for them. Second, if a test that kills only when the whole suite runs ever shows up asFLAKY, fix that test's isolation.Taken as working rulings (the Maintainer may overturn them in review):
FLAKYfails the required check: reporting it and passing would let a rule through unchecked, which is what this story exists to stop.