Skip to content

Mutation run: a shard whose baseline fails names the tests that failed, and tests.test_tools.Trials is brought down from 463 s #438

Closed

Description

@don-petry

Why. Two costs found while landing #437.

  1. A shard's baseline failure prints only "the tests fail before anything is broken", with no test name. To find the cause, a builder has to reproduce the shard at a desk. On a PR waiting for its record's place in line, this hid which failures were the expected numbering gap (Each job the Factory writes stops its own work under a step limit, so a job too long fails and is never cancelled (#388, record 0098) #434 had to be read from logs).
  2. tests.test_tools.Trials takes 463 s on a desk, unmutated (33 tests, read 2026-10-30). A full make verify now takes about 24 minutes, and AGENTS.md still says half a minute. Mutation run: a mutant names the method that kills it, never a class too slow to run whole #437 stops mutants from naming that class whole, but it remains the largest part of every verify and of the stop hook's 579 s timeout.

Done when

  • A failing baseline prints the failing tests' names, each with its first line of error, within a bound that keeps the log readable.
  • Trials is measured test by test, and what is slow is named. Bring it down where the time is not what the test asserts, without dropping a case.
  • AGENTS.md's Verify section states the real time, so tools/check_context.py's reader isn't misled.
  • Say whether the stop hook should keep running a whole verify, given that it times out on every builder's turn.

Follows #398, #399 and #437.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions