Skip to content

Mutation run: a kill is confirmed, and one copy per worker - #153

Merged
don-petry merged 6 commits into
mainfrom
tools/mutant-kills-confirmed
Oct 18, 2026
Merged

don-petry merged 6 commits into
mainfrom
tools/mutant-kills-confirmed

Conversation

@don-petry

@don-petry don-petry commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #147. Closes #152.

Why. Two problems with the mutation runner:

  • False kills: a mutant counted as killed when any test failed, even one failing for its own reasons under load. On CI: a Lens's verdict and the secret scan's rules run from the base #145 four mutants looked killed locally and survived in CI.
  • Disk churn: every mutant copied the whole repo into a fresh directory. On 2026-10-17, local runs across several sessions wrote about 550 GB in 12 hours. The file-event service grew to 42.6 GB, swap ran out, and the Mac had a kernel panic.

What changes

  • A kill is confirmed. The first failing test is run again alone on the mutant, and alone on the unbroken code in the same tree.
    • It must fail on the mutant and pass on the unbroken code.
    • If not, the whole suite runs once more on the mutant. A pass is SURVIVED; a second unconfirmed failure is FLAKY, which fails the run (a working ruling).
    • A timeout or crash stays ERROR.
  • Timing-sensitive tests: six tests are marked @wall_clock and skipped during the mutation run; they still run in make verify. A test holds that the marked set is exactly the set of tests that read the clock, by file, class and method, including from time import and import time as.
  • One copy per worker (Mutation run: one copy per worker, not one per mutant #152, folded in at the Maintainer's call):
    • Each worker copies the tree once, 2.8 MB and 227 files, with no .git and no caches. It runs mutants one at a time in that copy.
    • A mutant writes only the file it breaks. However the run ends, crash or hang included, the original bytes are written back and read again before the next mutant. A copy that doesn't read back is an ERROR, and the copy is never reused.
    • Each mutated file gets its own timestamp, so no bytecode is reused across mutants.
    • At most 4 workers by default.
  • Test ids: tests are discovered from the repo top (-s tests -t .), so a failing test's id can be run alone. This matches tools/verify.py.
  • Records:
    • Decision 0027 records it, tools/AGENTS.md names its test, and lesson L-60 is added.
    • The shard-time comment in verify.yml and its two copies is updated. The figure is an estimate until CI shows it.

Verified:

  • make verify passes (524 tests) on a plain git archive export with PATH=/usr/bin:/bin; the record was renumbered in the export only, for the 0027 gap.
  • The runner's 18 tests pass, including a test that a mutant leaves no trace in the next mutant's tree, and one where a test that fails without .git comes out FLAKY.
  • Five named mutants with 2 workers: 5 of 5 killed in 101 s, including the one that skips the restore.
  • No full local mutation run: it is banned on this Mac until this merges. CI's six shards are the check.

Reviewed by: a fresh-context Claude review, a security pass, and AGY (Gemini 4 Argon) on 9abeffd. All their findings are fixed in 4f31038:

  • the unbroken-side rerun ran in the checkout rather than an export, so a change could fake its own kills;
  • a multi-test only case was untested;
  • three rules had no mutant;
  • the clock-mark test checked a subset;
  • a stale AGENTS.md row and the time comment.

The copy-once rework in d163dba was read again by a focused Claude review and by AGY (Gemini 4 Argon), and AGY approved with no findings. The review found no high or medium bugs. Its three low findings all fail safe:

  • Only the mutated file is checked: a file a test leaves elsewhere in the copy is not detected. That gives FLAKY or ERROR, never a false kill, and the suite looks hermetic today.
  • A second Ctrl-C can delete the worker directory while suites still run.
  • The restored timestamp is not read back; the bytes are.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • Testing
    • Mutation checks now verify that failures are caused by the change being tested, reducing false positives from flaky or load-sensitive tests.
    • Timing-sensitive tests are skipped during mutation checks. Unconfirmed failures and test errors fail the run.
  • Documentation
    • Updated mutation-check timing estimates to about 450 mutants per shard, with a six-minute shard estimate. Six shards are still expected to handle about 800 mutants in under 15 minutes.

@don-petry
don-petry requested a review from a team as a code owner October 18, 2026 00:37
@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 26 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: combsmith/software-factory/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: c776c1f4-5934-45bb-a082-990a68a5babd
📥 Commits

Reviewing files that changed from the base of the PR and between 0043c68 and 5f1fb2b.

📒 Files selected for processing: 2
  • tests/test_tools.py
  • tools/mutants.py

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration
  • Configuration used: Repository: combsmith/software-factory/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 58fb045d-40a5-4a9e-8445-2c8bb338ae27
📥 Commits

Reviewing files that changed from the base of the PR and between ab22a1e and 0043c68.

📒 Files selected for processing: 10
  • .github/workflows/verify.yml
  • docs/decisions/0027-a-mutant-is-killed-by-a-test-that-fails-on-it-twice-and-passes-without-it.md
  • docs/lessons.md
  • docs/standards/workflows.md
  • examples/workflows/verify.yml
  • tests/support.py
  • tests/test_hooks.py
  • tests/test_tools.py
  • tools/AGENTS.md
  • tools/mutants.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

📝 Walkthrough

The mutation runner now confirms failures by rerunning the named test on mutant and unbroken code. It reuses worker copies, limits concurrency to four workers, and skips tests marked as wall-clock-dependent during mutation runs. Tests and documentation describe the updated behavior and runtime estimate.

Changes: Mutation runner

Layer / File(s) Summary
Failure attribution and confirmation
tools/mutants.py, tests/test_tools.py, tools/AGENTS.md, docs/decisions/0027-a-mutant-is-killed-by-a-test-that-fails-on-it-twice-and-passes-without-it.md, docs/lessons.md
The runner identifies a single failing test and reruns it on the mutant and unbroken code. A kill requires failure on the mutant and a pass on unbroken code. Unconfirmed failures, timeouts, and unreadable test output receive separate outcomes. Tests cover these cases.
Worker-copy restoration and reuse
tools/mutants.py, tests/test_tools.py
Workers share repository copies. The runner verifies and restores changed files before reuse, replaces copies marked as broken, and caps concurrent workers at four. Tests cover copy reuse, restoration, exceptions, and timeouts.
Wall-clock test handling and runtime guidance
tests/support.py, tests/test_hooks.py, tests/test_tools.py, .github/workflows/verify.yml, examples/workflows/verify.yml, docs/standards/workflows.md
The wall_clock decorator skips marked tests when mutation mode is active. Timing-sensitive tests use the decorator. Workflow guidance now estimates about 450 mutants at six minutes per shard and says each kill is confirmed by two runs of one test.

Priority: ➖ Normal

Estimated code review effort: 4 (Complex) | ~45 minutes

Change: Bug fix · Severity of issue fixed: Medium

Sequence Diagram(s)

sequenceDiagram
  participant Attempt as tools.mutants.attempt
  participant Suite as tools.mutants.suite
  participant Confirm as tools.mutants.confirm
  Attempt->>Suite: Run mutant suite
  Suite-->>Attempt: Return failing test output
  Attempt->>Confirm: Identify and confirm the failed test
  Confirm->>Suite: Run named test on mutant
  Suite-->>Confirm: Return mutant test result
  Confirm->>Suite: Run named test on unbroken code
  Suite-->>Confirm: Return unbroken test result
  Confirm-->>Attempt: Return kill, unconfirmed, or error
Loading
Merge Risk: ⚪ Minimal · up to 0043c

The change tightens how mutation kills are confirmed and reuses worker copies. No concrete merge-blocking risk was found. CI's shards remain the check for the full mutation run.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Linked Issues check Warning #147 is addressed. confirm() reruns the failing test on the mutant and on restored code. The tests cover unrelated failures, unbroken-code failures, flaky results, errors, test-id parsing, and `@wal… Add an automated test that measures or bounds total bytes copied for N mutants and fails if the runner copies a full tree per mutant. Retain the existing copy-reuse and restore checks.
Docstring Coverage Warning Docstring coverage is 26.19% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 42 functions across 4 files. (6 skipped: … Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 3 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title clearly and concisely describes the two primary changes: confirmed mutant kills and one worker copy reused per worker.
Out of Scope Changes check Passed The changed runner code, tests, timing-test support, workflow comments, decision, lesson, and agent guidance all support #147 or #152. The evidence shows no unrelated functional change.
Full details: Linked Issues check

Explanation

#147 is addressed. confirm() reruns the failing test on the mutant and on restored code. The tests cover unrelated failures, unbroken-code failures, flaky results, errors, test-id parsing, and @wall_clock exclusions. #152 is partly addressed. Trees creates one copy per worker, restores and reads back the changed file, rejects changed copies, and defaults to at most four workers. The reviewed evidence does not establish the required automated byte-bound test for N mutants.

Resolution

Add an automated test that measures or bounds total bytes copied for N mutants and fails if the runner copies a full tree per mutant. Retain the existing copy-reuse and restore checks.

Full details: Docstring Coverage

Explanation

Docstring coverage is 26.19% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Analyzed 42 functions across 4 files. (6 skipped: 6 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

✨ Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-18T01:36:04.594065Z 5f1fb2b New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@don-petry
don-petry force-pushed the tools/mutant-kills-confirmed branch from d163dba to bfd031c Compare October 18, 2026 00:47
@don-petry

Copy link
Copy Markdown
Contributor Author

Adversarial Review (AGY - Gemini 3.8 Flash High)

Evaluated against config/rubric-pr-review.json for commit bfd031c:

Rubric Evaluation

  • PR-1 (Requirements & Scope): PASS
    • Fully implements Issue #147 and Issue #152, rebased cleanly on latest main (commit b99f3bf):
      • Kill confirmation: reads failing test id, re-runs alone on the mutant and alone on unbroken code. Requires failing on the mutant and passing on unbroken code to count as killed.
      • Handles unconfirmed failures: re-runs suite; passes report as SURVIVED, while second unconfirmed failure reports as FLAKY (which fails the check run).
      • Timing-sensitive tests: 6 wall-clock dependent tests marked @wall_clock and skipped under MUTATING to avoid load-induced false kills.
      • Worker optimization (Issue Mutation run: one copy per worker, not one per mutant #152): copies tree once per worker without .git or caches; restores only the single mutated file with updated timestamps between mutants; caps default concurrency at 4 workers.
      • Test runner discovery aligned with tools/verify.py (-s tests -t .).
  • PR-2 (Security & Execution Safety): PASS
    • Tooling and test changes only; zero writes outside permitted effects.
  • PR-3 (Regression & Functional Tests): PASS
    • Exhaustive test suite added to tests/test_tools.py::Mutants covering kill confirmations, unconfirmed / flaky failures, copy isolation, and clock-mark completeness.
    • Dedicated mutants added in tools/mutants.py verifying all confirmation rules and worker safety invariants.
  • PR-4 (Kernel Purity): PASS
    • Zero modifications to kernel/.
  • PR-5 (Documentation & Invariants): PASS
    • Decision 0027 added with full Boundaries table mapped to tests. Decisions 0001 through 0027 form a zero-gap contiguous sequence.
    • Lesson L-60 documented in docs/lessons.md; tools/AGENTS.md and workflow standards updated.

Findings

  • Critical: 0
  • High: 0
  • Medium: 0

Verdict: PASS

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: d163dba3cf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/mutants.py Resolved
Comment thread tools/mutants.py Resolved
Comment thread tests/test_tools.py Resolved
@don-petry

Copy link
Copy Markdown
Contributor Author

Adversarial Review (AGY - Gemini 3.8 Flash High)

Evaluated against config/rubric-pr-review.json for commit b47e1f7:

Rubric Evaluation

  • PR-1 (Requirements & Scope): PASS
    • Accurately addresses review feedback on wall-clock test detection:
      • Expands wildcard imports (from time import *) to all clock names in test_every_test_timed_by_the_wall_clock_is_marked_and_skipped_when_mutating, ensuring any timing-sensitive tests using wildcard imports cannot bypass the @wall_clock requirement.
      • Zero out-of-scope changes.
  • PR-2 (Security & Execution Safety): PASS
    • Test-only modification; zero writes outside permitted effects.
  • PR-3 (Regression & Functional Tests): PASS
    • test_every_test_timed_by_the_wall_clock_is_marked_and_skipped_when_mutating verified passing.
    • tools/check_decisions.py and tools/check_context.py exit clean (0).
  • PR-4 (Kernel Purity): PASS
    • Zero modifications to kernel/.
  • PR-5 (Documentation & Invariants): PASS
    • Decision 0027 boundary invariants preserved.

Findings

  • Critical: 0
  • High: 0
  • Medium: 0

Verdict: PASS

don-petry and others added 4 commits October 17, 2026 20:00
A failing test counted as a kill, whatever made it fail, so a test that
failed under load passed four rules nothing checked as checked.

Now the one test that failed runs alone again on the mutant, where it
must fail, and alone on the unbroken code, where it must pass. A failure
short of that runs the suite once more: a pass is SURVIVED, a confirmed
failure is killed, and a second unconfirmed one is FLAKY, which is not
counted and fails the run. A timeout on any run stays an ERROR.

The test is read from the line unittest prints under its rule of =, in
the 3.9/3.10 and 3.11+ forms, a subtest's and setUpClass's among them.
The suite is now discovered from the top of the copy, as verify.py does,
so a test's id is its dotted path and can be run alone.

Tests timed by the wall clock are marked @wall_clock, skipped when
mutating and run as they are in make verify; a test holds that each one
that reads the clock or sleeps carries the mark.

Decision 0027, lesson L-60, and eleven mutants of the runner's own rules.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or each rule

From the review, the security pass and a second model's read.

- The unbroken side of a confirmation runs in a second copy, made as
  the mutant's is. The checkout has a .git and is not under the system's
  temporary directory; a test that fails only in a copy failed on every
  mutant and passed unbroken, so a change could make its own mutants look
  killed. A test holds that such a test kills nothing.
- A test named to run alone is run alone in a suite of more than one.
- Mutants for the rules that had none: the copy, `only`, a confirmation
  that errors, a failure line that cannot be read, and a subtest suffix.
- The @wall_clock test asserts the marked set equals the timed set, by
  file, class and method, and reads `from time import` and `import time
  as` too.
- The tools/AGENTS.md row names a test that checks what it says.
- The shard time in verify.yml and its two copies.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every mutant copied the whole tree into a fresh directory, and the last
round added a second copy for each kill. Over hundreds of mutants and
several runs that wrote hundreds of gigabytes and stopped the machine.
Closes #152.

- Each worker copies the tree once and is lent to one mutant at a time.
  A mutant writes only the file it breaks, with a time of its own past
  the file's so no bytecode compiled from another is taken for it, and
  however its runs end, a crash or a hang among them, the file's own
  bytes and time are written back and read again.
- The unbroken side of a confirmation runs in the same copy with the
  file put back: an export on both sides, so .git is absent on both.
- A copy whose file does not read back, or that is lent with a file
  that is not the tree's, is an ERROR for that mutant and never lent
  again.
- At most four workers unless told.
- A test holds that a mutant leaves no trace for the next, and mutants
  hold the restore, the guard, the one copy and the unbroken side.

One copy is 2.8 MB, 227 files, with no __pycache__ and no .git.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry force-pushed the tools/mutant-kills-confirmed branch from b47e1f7 to 0043c68 Compare October 18, 2026 01:01
@don-petry

Copy link
Copy Markdown
Contributor Author

Adversarial Review (AGY - Gemini 3.8 Flash High)

Evaluated against config/rubric-pr-review.json for commit 0043c68:

Rubric Evaluation

  • PR-1 (Requirements & Scope): PASS
    • Rebased cleanly onto latest main (commit ab22a1e, incorporating PR #150 and PR #151):
      • Fully implements Issue #147 and Issue #152.
      • Kill confirmation requires single-test rerun on mutant and unbroken code; unconfirmed failures rerun suite; second unconfirmed failure reports as FLAKY (failing run).
      • Timing-sensitive tests marked @wall_clock and skipped under MUTATING; wildcard from time import * imports covered.
      • Worker copy-once architecture restores only single mutated file with fresh timestamp; max 4 workers by default.
  • PR-2 (Security & Execution Safety): PASS
    • Zero writes outside permitted effects.
  • PR-3 (Regression & Functional Tests): PASS
    • 18 comprehensive tests in tests/test_tools.py::Mutants.
    • tools/check_decisions.py and tools/check_context.py exit clean (0).
  • PR-4 (Kernel Purity): PASS
    • Zero modifications to kernel/.
  • PR-5 (Documentation & Invariants): PASS
    • Decision 0027 added with full Boundaries table. Contiguous ADR sequence 0001 through 0027 preserved without gaps.
    • Lesson L-60 added; tools/AGENTS.md and workflow standards updated.

Findings

  • Critical: 0
  • High: 0
  • Medium: 0

Verdict: PASS

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Oct 18, 2026
Merged via the queue into main with commit a526487 Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutation run: one copy per worker, not one per mutant Mutation run: a mutant is killed only by a test that fails because of it

1 participant