Skip to content

Mutation run: a mutant is killed only by a test that fails because of it #147

Closed

Description

@don-petry

Part of the epic "Epic: slice 0, foundations. The repo holds the line, its context is tested, and the platform facts are recorded".

Why. On "CI: a Lens's verdict and the secret scan's rules run from the base" (#145), a local python3 -m tools.mutants reported four mutants killed that CI's shards reported SURVIVED. The code they mutated had become dead; no test could kill them. Locally they were "killed" by an unrelated test failing under load (the Mac's load average was near 30 with several agents running): tools/mutants.py counts any failing test as a kill. So a flaky or timing-sensitive test failing for its own reasons hides a rule nothing checks — the exact thing the mutation check exists to catch.

Done when

  • A mutant is killed only when a test fails for that mutant: the unbroken suite's failures, and any test that fails on a re-run of the unbroken code, do not count (for example: a kill must be reproduced, or must come from the tests the mutant names first, or a timing test is excluded from the mutation run).
  • Timing-sensitive tests (a deadline in the hooks, a budget) are marked and kept out of the kill count, or made deterministic.
  • A test holds that a mutant whose only failure is an unrelated test is reported SURVIVED.

Activity

  1. don-petry commented on Oct 17, 2026

    @don-petry
    ContributorAuthor

    Working rulings (overturnable in review)

    Solution Architect, delegated. On main at 3d4109c (#145). 404 mutants today, 16 with named tests.

    One correction first. A timeout is not a kill today. verdict() makes it an ERROR, which fails the run. The false kills in #145 came from a failing test: each run stops at its first failure (-f), and any failure counts.

    1. What a kill is. A kill is a test that fails on the mutant, fails again when run alone on the mutant, and passes when run alone on the unbroken code.

    • attempt() reads the one failing test from the output. It reads both forms of the line: FAIL: test_x (tests.m.T) (3.9, 3.10) and FAIL: test_x (tests.m.T.test_x) (3.11 and later, which CI runs). It also reads a subtest suffix and setUpClass, which names the class. A line it cannot read is an ERROR.
    • It runs that test twice more: run(work, only=id) and run(root, only=id).
    • If either run disagrees, the failure is unconfirmed. The whole suite then runs once more on the mutant. A pass is SURVIVED, and a confirmed failure is killed. A second unconfirmed failure is FLAKY.
    • The unbroken suite's first run stays as it is. If it fails, the check stops before any mutant.

    Why: only re-running on both sides tells three failures apart: one this mutant causes, one already broken, and one caused by load. In #145 the re-run on the unbroken code alone would still have passed. The cost is two single-test runs a mutant, about 35 s a shard. The "tests first" field stays, only for speed.

    2. Timing-sensitive tests. They are marked @wall_clock in tests/support.py, which skips them under MUTATING as @pending does. They still run in make verify with their margins as they are.

    • test_hooks.py: Identity.test_a_check_that_runs_out_of_time_refuses, test_the_brace_bomb_is_answered_ask_within_the_deadline, test_a_brace_left_open_is_read_within_the_budget and test_a_wildcard_that_would_backtrack_is_read_within_the_budget.
    • test_tools.py: test_a_run_that_hangs_is_ended_reported_and_fails_the_check and test_mutants_are_reported_in_their_order_however_their_runs_finish.
    • Not marked: test_a_parse_over_its_budget_asks and RulesChanged's seconds=0 case, since load only makes a spent budget more certain. The 60 s subprocess timeouts are not marked either; the confirmation covers them.

    Why: no mutant touches a hook's budget, so skipping these tests loses no kill. A test holds the mark.

    3. Timeouts. A timeout stays an ERROR, on any run, including a confirmation. It fails the run. It is never killed and never FLAKY. Why: 600 s is 10 to 30 times a full run, so a timeout means a hang. A rule that only a hang shows needs a test that fails.

    4. Reporting. FLAKY <mutant> (<test id> passed when run again on the mutant | failed on the unbroken code too; mark or mend that test, then run python3 -m tools.mutants again). FLAKY fails the run as ERROR does, and the summary line does not change. Why: a mutant nobody showed is killed is a rule unchecked. Passing it would bring back #145 in a quieter form.

    5. Tests, mutants, record.

    • Tests in test_tools.py::Mutants. The issue's case: test_a_mutant_whose_only_failure_is_unrelated_survives, once with told() and once against a real suite_of whose test fails on its first run only. Also tests for:
      • a confirmed kill;
      • a test that fails unbroken;
      • two unconfirmed failures, which are FLAKY;
      • each Python's form of the failure line;
      • every test that reads the clock carrying the mark.
    • Mutants (on mutants.py, with the text split in two as the shard mutants are, and the new tests named as their fifth part):
      • the re-run on the mutant dropped;
      • the re-run on the unbroken code dropped;
      • an unconfirmed failure taken as a kill;
      • FLAKY counted as killed;
      • @wall_clock not skipping under MUTATING.
    • Docs. Change the tools/AGENTS.md row "Only a failing test kills a mutant" to say the above. Add a lesson, "A test that failed under load killed a mutant", with these checks.
    • Record: docs/decisions/0027-a-mutant-is-killed-by-a-test-that-fails-on-it-twice-and-passes-without-it.md, new, replacing nothing. It records the alternatives not taken: named tests only, and no -f.
    • Follow-ups. First, let Budget in protected.py and the clock in identity.py take an injected clock, and add mutants for them. Second, if a test that kills only when the whole suite runs ever shows up as FLAKY, fix that test's isolation.

    Taken as working rulings (the Maintainer may overturn them in review):

    • FLAKY fails the required check: reporting it and passing would let a rule through unchecked, which is what this story exists to stop.
    • The confirmation runs add about 35 s a shard, against a run of about 10 minutes.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions