Repository navigation
The advisory check of a changed line of Python no test notices: untested.yml and tools/untested.py (record 0166) - #634
Conversation
…ted.yml and tools/untested.py (record NNNN) Each hunk of Python a pull request changes outside tests/ is reverted alone, and the head's tests run on it through the mutant run's own attempt. A crash is never a kill; an edit that cannot run falls back to its file's revert, then, for a hunk adding more than 40 lines, to mutants of them. A trailer Behaviour-preserving: <path>:<start>-<end> covers a survivor. For its first month the job reads no merge group, is in no job's needs, and exits 0 whatever it finds (issue 589's rulings of 2026-11-07). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Important Review skippedAuto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the
|
Codex Review Summary: ❌ FailedThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 347315feae
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ile's kill is counted apart, and the runner's files are held from the head From the security pass and the Claude review of 347315f, and Codex's thread: - the mutant lists' data is an entry of constants, names and helper calls, or a blank or a comment line; a hunk touching code among them is read; - a kill of the file's revert is counted apart, killed_by_file, never as the hunk's own; - a trailer is held to every hunk of the change, those never run among them; - a large hunk whose every mutant the budget left is UNCHECKED over budget; - tests for a hunk that only removes, the parse and import guards, the 40-line limit, tests/ left out, and the runner's files taken from the head's environment, each with its mutant; - a condition is negated at the AST's offsets, not by a regex that backtracks (SonarCloud). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n its workflow and its record Argon's read of 347315f: the job runs the base's tool, which the base does not have until this merges. Kept, as decision 0023 gives it; the job is in no job's needs, so it blocks nothing. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 64a33e547e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s is data, and each fallback tried is counted Codex's read of 64a33e5: - a line on which any part of the list that is no literal entry starts or ends is never data, so a comprehension's clause or a condition beside an entry is read; an entry may index an item, as ALSO_AT[0]; - fallback_file and fallback_mutants count each fallback tried, whatever edit gave the outcome. Each with its test and its mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of minus infinity for the spent budget The mutant shards' baseline refused them: a test that reads the clock is marked and skipped when mutating (test_every_test_timed_by_the_wall_clock_is_marked_and_skipped_when_mutating). These read it only to set a deadline. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: c5dec7670f
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… a kill's confirmation among them Codex's read of c5dec76: the two runs that confirm a kill could each start past the budget with a whole TIMEOUT, and outlast the step. Each run is now cut to what is left before the budget's end and one TIMEOUT, and one past it is over budget. With its test and two mutants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: a484f99dc8
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…s made of an annotation never evaluated Codex's read of a484f99: - an import that hangs takes its whole 60 s, so neither the head's probe nor an edit's begins past the budget; - a fallback mutant Python reads as the head, as of a deferred annotation, is never made. Each with its test and its mutants. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: ad01122c95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…e endings, and calls a hung mutant an ERROR Codex's read of ad01122: - reading the whole suite imports every test module under a TIMEOUT of its own, so it does not begin past the budget; - the head's file is read as its bytes, so a CRLF file's edit is not STALE; - a large hunk any mutant of which hung, or would not run, is an ERROR, never killed by its neighbours. Each with its test and its mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt, and an edit's import probe The mutant shards at 6f36c4a: "a run of the tests begins past the budget" survived, since the bound at the budget's end also refused the run; and "an edit's import is probed past the budget" survived, since a spent budget skips the head's probe and so every edit's. Each now has a test that sees it alone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 95998a6a95
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
… UNCHECKED, never killed Codex's read of 95998a6: one mutant killed and five left by the budget made the hunk killed. It is now UNCHECKED, "over budget", unless a mutant survived, was flaky or hung. With its case and its mutant. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: 24d7e6890e
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
…ng name, and never counts a trailer's span Codex's read of 24d7e68: - a line is ended by a newline alone, so a form feed or U+2028 in a string no longer moves a revert onto the wrong line and makes a real change look the same; - a crash is read from unittest's whole output, since the mutant run's failures cut a long test's name before its exception; - a trailer is checked against the hunks' lines without making a set of every line in its span. Each with its test, its mutant and its Boundaries row. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… that now reads it The trailer's span is now read without a set of its lines, so "a trailer that names no changed line stands" went STALE. It mutates the new condition. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
…eback mutant's text Shard 1 at cfb6b30: "the traceback's first line is read as the error" went STALE, since the new mutant's text held its target a second time in tools/mutants.py. The new one's string is split as the old one's is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
…ER; 0167 counts eleven Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>



This adds the advisory check of issue 589, "A changed line of Python that no test notices fails verify, unless the change says it keeps behaviour". It follows the design note and the Factory Builder's rulings of 2026-11-07. It fails no pull request: for a month from its merge it reports, and this lane posts a weekly list to the issue.
What the Maintainer was asked
.github/workflows/untested.ymlis a new file under a protected path. The Maintainer approved it on 2026-11-08 ("Approve (Recommended)"), running on bothpull_requestandmerge_group.verify'sneedsor a second ruleset context, and the trade-off of each against the emergency fix (design §9.1).untestedrun will fail. The job runs the base'stools/untested.py, and the base has none yet. For the same reason this PR's merge-group run fails once too: it checks outmerge_group.base_sha, which is main without the tool. It is in no job'sneeds(verifyneedschecks,commitsandmutants), and main requires onlyverify, so it blocks nothing. Every later PR runs it, and every later merge group declines it.enqueue_clean.shwaits for every check to be green, so this PR can't pass it. With the Factory Builder's go-ahead of 2026-11-08, it was enqueued directly withenqueuePullRequestat its reviewed head, once every other check and the merged-tree verify were green.What it does
The job checks out the base and runs
python3 -m tools.untested. The tool:git archive;git diff -U0from the merge base, in a*.pyoutsidetests/,trials/,examples/andschemas/, leaving out the entries of theISSUEandMUTANTSlists oftools/mutants.py, but not the code among them;from __future__ import annotations);tools/mutants.py's ownattempt, so a kill is confirmed by two runs of the one test that failed.CRASHSyntaxError,ImportError,NameError(and kin) or a test module that will not import notices itkilled_by_file), then, for a hunk adding more than 40 lines alone, to at most 6 mutants of those lines (ruling 1). A mutant that hangs makes the hunk an ERROR; one the budget left makes it UNCHECKEDTIMEOUTafter it, so the job ends inside its stepThe tests run first:
tools/: each test class that reaches the module (ruling 2), and for a class inWHOLE_TOO_LONG, its methods.The trailer.
Behaviour-preserving: <path>:<start>-<end>, read from the head commit by git's own parser, covers a survivor or an UNCHECKED hunk when it holds every line the hunk added. A trailer naming no changed line, or a file the change leaves alone, is a problem.Advisory, by construction:
untested.ymlruns onpull_requestandmerge_group.check_workflowsrequires both of any workflow onpull_request, so on a merge group the tool says it did not run, and exits.untested/1 {counts}, is what the weekly list is built from: hunks, killed, survived, survived that only remove, unchecked, over budget, flaky, error, file and mutant fallbacks, covered, trailers and their problems, and seconds.Measured on this desk
9ca55015), kernelfb601dba),tools/check_workflows.py5c4602d3),tools/hooks/shell.pyRecords
0166, "A changed line of Python no test notices is reported for a month before it can fail". The Factory Builder gave the number on 2026-11-08: #628 took 0164 and #632 took 0165. Twenty-three Boundaries rows, each naming its tests in
tests/test_untested.pyand its mutants.Checks
tests.test_untestedcfb6b30; all 12 of CI's shards pass ata65ddb0make verifyon an export ofa65ddb0: record 0166, main merged in atf1e3c7ba65ddb0untested, its one-time run, as aboveReads
347315f347315f347315f347315fto24d7e68Muse was not run: it is out of quota until 2026-11-09T00:00Z, and its read is owed then. Codex reached its usage limit at
cfb6b30and gave no read of it.a13632b, each with a test and a mutanta13632ba13632b, then553edaefor some of thema13632bMUTANTSassignment counts as dataa13632b,359cb79a13632b:killed_by_fileuntestedrun fails359cb79a484f99,ad01122,6f36c4aad01122,6f36c4aa13632b793ba05,cfb6b30, each with a test and a mutanta65ddb0🤖 Generated with Claude Code