Skip to content

Mutation run: a shard's baseline is its share of what every mutant names, and the whole suite runs in dealt parts (record 0093) - #412

Merged
don-petry merged 5 commits into
mainfrom
claude/ci-398-mutant-shares
Oct 30, 2026
Merged

don-petry merged 5 commits into
mainfrom
claude/ci-398-mutant-shares

Conversation

@don-petry

@don-petry don-petry commented Oct 29, 2026 •

Copy link
Copy Markdown
Contributor

Closes #398. Record 0093, which extends 0083 without editing it. Synced with main after 0092 (#408) landed, so the records' gap check passes.

Why. #399 (record 0083) made each shard's baseline the tests its own mutants name, dealt among four workers, and accepted two gaps. Mutants name whole classes, and the shards share them, so each shard still named about 840 to 910 of the 1,128 named tests, and its baseline grew nearly as the suite does. A mutant whose named tests pass still ran the whole suite as one run held to 600 s, which took 523 s on a desk.

What changes (tools/mutants.py, its tests, tools/AGENTS.md's row and record 0093; the workflow is unchanged)

  • A shard's baseline is a disjoint share. It is of the tests every mutant names, read method by method in its tree as 0083 reads them: each count-th from the shard's place. Across the shards each named test runs once. With eight shards each runs 141, where it ran about 840 to 910, and a share shrinks as shards are added. A share may hold none, and then nothing runs.
    • A test red on the unbroken code fails the one shard whose share holds it, as red, before any mutant is broken, whichever shard's mutants name it.
    • A run with no shard, filtered or at a desk, runs what its own mutants name, as before.
    • Where any name cannot be read method by method, one shard's list cannot be told to be another's, so each shard runs every name, read as far as it can be: no named test is left in no share.
    • selected gives the share as a fourth value. The front door is now front(), which hands it to main, so a test can drive it.
  • The whole suite runs in dealt parts. Where a mutant's named tests pass, its tests are dealt in turn into as few parts as hold at most PART_MOST, 100, each. Each part keeps the suite's order, the slow modules last.
    • Each part is one run held to the 600 s, one after another in the mutant's copy, and the next runs only when the last passed. The same holds for the second whole-suite run after an unconfirmed failure.
    • confirm's runs are unchanged: one test, or one class where its setup failed.
    • A module that cannot be read method by method is a part of its own.
    • PART_MOST is held at 100 by a test.
    • Dealt, not cut in order: tests.test_tools.Trials holds tests of 97, 72 and 31 s side by side. On a loaded desk (940 s for the suite), consecutive parts of 250 reached 420 s and of 50 still 306 s. Dealt parts of 100 made 14 parts of 9 to 124 s.

Measured. The whole suite on this desk, unbroken, in its 14 dealt parts of 93 or 94 tests: every part passed, the parts took 43 to 51 s each where measured. The shares: 141 tests in each of 8 shards.

Checks (on ed97cad, after the sync and the fix round below)

  • make verify: every check passes, and 1,332 tests run OK.
  • python3 -m tools.mutants: 3,224 of 3,225 killed at this desk. The one left, "a signal leaves the reader's token live" (tools/trial_reader_reads.sh), survives on main too at this desk, whose /bin/sh is macOS's; it is not this change's, and CI's shards are the gate.
  • The new and re-pointed mutants of this change, each applied by hand with its named tests: all killed.
  • CodeQL: no workflow changes.

Reviews after the sync, structured (review-findings/1, parsed by review_parse.py; head 820e8ce)

Reviewer Head Unread parts Blocking Major Minor Info
agy 820e8ce none 0 0 0 0
muse 820e8ce none 0 1 0 0
claude-review 820e8ce none 0 1 1 1
claude-security 820e8ce none 0 0 0 0
Finding Severity Criterion Where What Disposition
muse#1 major PR-1 tools/mutants.py:9428 Whole-suite parts cannot run a module whose tests are not all TestCase methods fixed in ed97cad: a name whose tests are not all TestCase methods by their own ids runs whole
claude-review#1 major PR-5 tools/AGENTS.md:35 The changed row now says every test runs in parts, but neither test it names fails if the parts are broken fixed in ed97cad: the row names test_the_whole_suite_runs_in_dealt_parts_each_held_to_the_limit
claude-review#2 minor none tools/mutants.py:9431 A test module with no test methods becomes its own part, which exits 5 on CI's Python 3.12 and makes the mutant an ERROR fixed in ed97cad: a module of no test runs nothing, with a test and a mutant
claude-review#3 info none docs/decisions/0093-a-mutant-shards-baseline-is-its-share-of-what-every-mutant-names-and-the-whole-suite-runs-in-dealt-parts.md:10 A red named test now makes every other shard that names it run its mutants and report them FLAKY, instead of stopping at the baseline as red fixed in ed97cad: record 0093 says the other shards end FLAKY

Reviews before the sync

  • Claude review (against config/rubric-pr-review.json): 2 majors, 2 minors, 3 nits; all fixed.
    • Major: the front door's share=share had no test or mutant. Now front(), driven by a test with a shard and with a filter, which also holds workers=at_once, a gap from before.
    • Major: nothing held PART_MOST = 100. A test asserts it, and a mutant breaks it.
    • Minor: a shard whose tree could not be read fell back to its own names, which could leave a test in no share and a red one read as FLAKY. Now every name runs in each such shard.
    • Minor: a module that cannot be listed in the whole suite made one unbounded run. It is now a part of its own, and 0093 says it is bounded by its own size.
    • Nits: 0093's fallback wording and the 834 to 903 figure; tools/AGENTS.md's row said the slow modules run last, which holds within each part.
  • Security pass: no blocker or major; 3 minors.
  • AGY (Argon): no findings (read 41819cf).
  • Muse (Wonder Full): 3 majors, 1 minor; all fixed. The tools/AGENTS.md row over 260 characters (fixed in b0eb541), and the same PART_MOST, front-door and fallback findings as above.

🤖 Generated with Claude Code

don-petry and others added 3 commits October 29, 2026 13:12
…mes, and the whole suite runs in dealt parts (record 0093)

Each shard's baseline is each count-th of the tests every mutant names,
method by method, so across the shards each runs once: 141 a shard of 1,128,
where each ran 834 to 903. A red test fails the shard that holds it. A tree
that cannot be read runs the shard's own names, as before.

A whole-suite run is dealt into parts of at most PART_MOST, 100, tests, each
one run held to the 600 s, the next only when the last passed, so no run
grows with the suite (issue 398).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…t parts, and the row stays under 260 characters

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ld, and a name that cannot be read is run in every shard and alone

The Claude review and Muse: the front door's share and PART_MOST's value had no
test or mutant; main() is now reached through front(), which a test drives with
a shard and a filter, and PART_MOST is asserted. The Claude review, Muse and the
security pass: a shard that could not read a name fell back to its own names,
which could leave a test in no share; any name that cannot be read method by
method now runs every name in each shard. In the whole suite, a module that
cannot be read is a part of its own. Record 0093 says so, and what bounds a run
of parts; tools/AGENTS.md's row says the suite runs in parts.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 29, 2026 19:03
@coderabbitai

coderabbitai Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 29, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-30T05:16:10.677858Z ed97cad New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

don-petry and others added 2 commits October 29, 2026 23:16
…s module runs whole, and the row names its test

Muse: a load_tests doctest's id is no test unittest runs by name; a name whose
tests are not all methods of a TestCase by their own ids is now kept whole. The
Claude review: a module that holds no test made a part of its own, which 3.12
exits 5 for; it now runs nothing. tools/AGENTS.md's row names the test of the
dealt parts, and record 0093 says a red test fails the other shards as FLAKY.
A test a module under tests/ takes from elsewhere is kept whole, so the check of
an id under tests/ keeps its mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Oct 30, 2026
Merged via the queue into main with commit 8b69f9a Oct 30, 2026
19 checks passed
@don-petry
don-petry deleted the claude/ci-398-mutant-shares branch October 30, 2026 07:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Mutant shards scale with their count: a disjoint share of the named tests for each baseline, and a survivor's whole-suite run kept under the limit

1 participant