Skip to content

Mutant shards scale with their count: a disjoint share of the named tests for each baseline, and a survivor's whole-suite run kept under the limit #398

Closed

Description

@don-petry

Part of the epic "Epic: slice 1, first approval. A Release on next reviews a pull request to this repo, and the machine user's approval stands" (#182).

Follows "A mutant shard's baseline runs what its own mutants name, so it no longer outgrows the 600 s a run is given" (#396). Its record, 0083, accepts this gap and links here.

Why. The fix for #396 runs each shard's baseline as the tests its mutants name, dealt among its four workers. It is far under the 600 s a run is given, but it does not scale with the shard:

  • Mutants name whole classes, and the shards share them, so each shard names 61 to 67 per cent of the suite's tests.
  • The baseline therefore grows nearly as the suite does, divided by four.

A second run is not covered at all. A mutant whose named tests pass, whether it survives or is killed by a test it does not name, still runs the whole suite, held to the same 600 s. The whole suite takes 523 s on a desk, and CI's runners are slower. Once it passes 600 s there, such a mutant is an ERROR, not SURVIVED or killed, and the shard fails for the run's length rather than for a rule. The Factory Builder, 2026-10-28: this may be needed within days.

What is wanted (the Factory Builder, 2026-10-28)

  • Each shard's baseline is a disjoint share of the tests every mutant names, so that across the shards each named test runs once, and a shard's baseline shrinks as shards are added.
  • A test red on the unbroken code still fails the mutation check, whichever shard holds it, and says so as red.
  • A surviving mutant's whole-suite run, and the runs confirm makes after it, are kept under the limit. The way may be splitting the whole suite among workers as the baseline is, or another bounded way. That run is never left to grow with the suite.
  • A test and a mutant hold each rule; if .github/workflows/verify.yml changes, tools/check_workflows.py holds its exact shape and CodeQL stays clean.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions