Skip to content

The advisory check of a changed line of Python no test notices: untested.yml and tools/untested.py (record 0166) - #634

Merged
don-petry merged 15 commits into
mainfrom
untested-advisory
Nov 8, 2026
Merged

don-petry merged 15 commits into
mainfrom
untested-advisory

Conversation

@don-petry

@don-petry don-petry commented Nov 8, 2026 •

Copy link
Copy Markdown
Contributor

This adds the advisory check of issue 589, "A changed line of Python that no test notices fails verify, unless the change says it keeps behaviour". It follows the design note and the Factory Builder's rulings of 2026-11-07. It fails no pull request: for a month from its merge it reports, and this lane posts a weekly list to the issue.

What the Maintainer was asked

  • .github/workflows/untested.yml is a new file under a protected path. The Maintainer approved it on 2026-11-08 ("Approve (Recommended)"), running on both pull_request and merge_group.
  • Nothing is made required. The switch stays open until the month ends (ruling 3). The record keeps both routes, verify's needs or a second ruleset context, and the trade-off of each against the emergency fix (design §9.1).
  • This PR's own untested run will fail. The job runs the base's tools/untested.py, and the base has none yet. For the same reason this PR's merge-group run fails once too: it checks out merge_group.base_sha, which is main without the tool. It is in no job's needs (verify needs checks, commits and mutants), and main requires only verify, so it blocks nothing. Every later PR runs it, and every later merge group declines it.
  • A one-time exception at enqueue. enqueue_clean.sh waits for every check to be green, so this PR can't pass it. With the Factory Builder's go-ahead of 2026-11-08, it was enqueued directly with enqueuePullRequest at its reviewed head, once every other check and the merged-tree verify were green.

What it does

The job checks out the base and runs python3 -m tools.untested. The tool:

  1. fetches the head and exports it with git archive;
  2. reads each hunk of git diff -U0 from the merge base, in a *.py outside tests/, trials/, examples/ and schemas/, leaving out the entries of the ISSUE and MUTANTS lists of tools/mutants.py, but not the code among them;
  3. skips a hunk whose module reads the same to Python without it (docstrings, comments, and annotations under from __future__ import annotations);
  4. reverts each remaining hunk alone, and runs the head's tests on it through tools/mutants.py's own attempt, so a kill is confirmed by two runs of the one test that failed.
Case Outcome
A test notices the revert, by an assertion or an exception outside CRASH killed
No test notices SURVIVED, with a warning on its lines and the trailer line it would need
The edit does not parse, its module will not import, or only a SyntaxError, ImportError, NameError (and kin) or a test module that will not import notices it Not a kill. It falls back to the file's every hunk reverted together, whose kill is counted apart (killed_by_file), then, for a hunk adding more than 40 lines alone, to at most 6 mutants of those lines (ruling 1). A mutant that hangs makes the hunk an ERROR; one the budget left makes it UNCHECKED
No edit of the hunk can be run, or the 20-minute budget ran out first UNCHECKED, with its reason. No run, import probe or read of the suite begins past the budget but a kill's confirmation, and none lasts past one TIMEOUT after it, so the job ends inside its step
Two failures that are not the edit's FLAKY

The tests run first:

  • For a hunk in tools/: each test class that reaches the module (ruling 2), and for a class in WHOLE_TOO_LONG, its methods.
  • Elsewhere: each test module that imports the module at any remove.
  • The slow simulator modules are left to the whole suite, which runs only when the first tests pass.

The trailer. Behaviour-preserving: <path>:<start>-<end>, read from the head commit by git's own parser, covers a survivor or an UNCHECKED hunk when it holds every line the hunk added. A trailer naming no changed line, or a file the change leaves alone, is a problem.

Advisory, by construction:

  • untested.yml runs on pull_request and merge_group. check_workflows requires both of any workflow on pull_request, so on a merge group the tool says it did not run, and exits.
  • It exits 0 whatever happens.
  • Its last log line, untested/1 {counts}, is what the weekly list is built from: hunks, killed, survived, survived that only remove, unchecked, over budget, flaky, error, file and mutant fallbacks, covered, trailers and their problems, and seconds.

Measured on this desk

Change Hunks Outcome Time
#607 (9ca55015), kernel 5 (4 in the mutant lists, 1 unchanged to Python) 5 killed 14 s
#606 (fb601dba), tools/check_workflows.py 12 (6 in the mutant lists) 4 killed, 8 UNCHECKED over budget; 5 went to their file's revert 1,262 s, the budget binding. The desk was running the mutant run beside it
#604 (5c4602d3), tools/hooks/shell.py 11 11 killed, 2 by their file's revert 75 s

Records

0166, "A changed line of Python no test notices is reported for a month before it can fail". The Factory Builder gave the number on 2026-11-08: #628 took 0164 and #632 took 0165. Twenty-three Boundaries rows, each naming its tests in tests/test_untested.py and its mutants.

Checks

Check Result
tests.test_untested 25 OK, 7 s
The 50 new mutants 50 of 50 killed at cfb6b30; all 12 of CI's shards pass at a65ddb0
make verify on an export of a65ddb0: record 0166, main merged in at f1e3c7b 1705 tests OK (expected failures 8)
CI at a65ddb0 Every check passes but untested, its one-time run, as above

Reads

Reviewer Head Blocking Major Minor Info
claude-review 347315f 0 1 5 1
claude-security 347315f 0 0 3 1
agy (Argon) 347315f 0 1 0 0
Codex (bot) 347315f to 24d7e68 0 0 14 (P2) 0

Muse was not run: it is out of quota until 2026-11-09T00:00Z, and its read is owed then. Codex reached its usage limit at cfb6b30 and gave no read of it.

Finding From Fixed in
A hunk that only removes is put back unchecked; tests/ left out, the parse and import guards and the 40-line limit held by no test claude-review a13632b, each with a test and a mutant
A trailer on a hunk never run is a false problem claude-review, Codex a13632b
A large hunk whose mutants all ran out of budget is not counted over budget claude-review a13632b, then 553edae for some of them
The runner's file-command variables reach the head's tests unheld by a test claude-security, claude-review a13632b
Code inside the MUTANTS assignment counts as data claude-security, Codex a13632b, 359cb79
A file revert's kill is counted as the hunk's own claude-security a13632b: killed_by_file
The head's tests can forge the log line the weekly list reads claude-security (info) The record says so; the switch must move the verdict out of the head's process tree
This PR's own untested run fails agy Kept, as decision 0023 gives it; said in the workflow and the record
A fallback tried but not giving the outcome is not counted Codex 359cb79
A kill's confirmations, an import probe, or the suite's discovery could outlast the step Codex a484f99, ad01122, 6f36c4a
A mutant of a deferred annotation; a CRLF file's edit is STALE; a hung mutant beside killed ones is killed Codex ad01122, 6f36c4a
A condition regex that backtracks SonarCloud a13632b
A form feed or U+2028 in a string moves a revert onto the wrong line; a long test name hides a crash; a trailer's span is counted line by line Codex 793ba05, cfb6b30, each with a test and a mutant
The new crash mutant repeated an older mutant's text, which went STALE in shard 1 CI a65ddb0

🤖 Generated with Claude Code

…ted.yml and tools/untested.py (record NNNN)

Each hunk of Python a pull request changes outside tests/ is reverted alone, and the head's tests run on
it through the mutant run's own attempt. A crash is never a kill; an edit that cannot run falls back to
its file's revert, then, for a hunk adding more than 40 lines, to mutants of them. A trailer
Behaviour-preserving: <path>:<start>-<end> covers a survivor. For its first month the job reads no merge
group, is in no job's needs, and exits 0 whatever it finds (issue 589's rulings of 2026-11-07).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner November 8, 2026 07:49
@coderabbitai

coderabbitai Bot commented Nov 8, 2026 •

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on this repository. Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

Run configuration

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • Trigger review
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

Share

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Nov 8, 2026 •

Copy link
Copy Markdown
Codex Review Summary: ❌ Failed

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ❌ Failed 2026-11-08T17:20:18.061586Z
We couldn’t complete this review. Comment @codex review to try again.
a65ddb0 New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 347315feae

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
don-petry and others added 2 commits November 8, 2026 02:00
…ile's kill is counted apart, and the runner's files are held from the head

From the security pass and the Claude review of 347315f, and Codex's thread:
- the mutant lists' data is an entry of constants, names and helper calls, or a blank or a comment line; a hunk
  touching code among them is read;
- a kill of the file's revert is counted apart, killed_by_file, never as the hunk's own;
- a trailer is held to every hunk of the change, those never run among them;
- a large hunk whose every mutant the budget left is UNCHECKED over budget;
- tests for a hunk that only removes, the parse and import guards, the 40-line limit, tests/ left out, and the
  runner's files taken from the head's environment, each with its mutant;
- a condition is negated at the AST's offsets, not by a regex that backtracks (SonarCloud).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…n its workflow and its record

Argon's read of 347315f: the job runs the base's tool, which the base does not have until this merges. Kept, as
decision 0023 gives it; the job is in no job's needs, so it blocks nothing.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 64a33e547e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
don-petry and others added 2 commits November 8, 2026 02:14
…s is data, and each fallback tried is counted

Codex's read of 64a33e5:
- a line on which any part of the list that is no literal entry starts or ends is never data, so a comprehension's
  clause or a condition beside an entry is read; an entry may index an item, as ALSO_AT[0];
- fallback_file and fallback_mutants count each fallback tried, whatever edit gave the outcome.
Each with its test and its mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…of minus infinity for the spent budget

The mutant shards' baseline refused them: a test that reads the clock is marked and skipped when mutating
(test_every_test_timed_by_the_wall_clock_is_marked_and_skipped_when_mutating). These read it only to set a deadline.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c5dec7670f

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
… a kill's confirmation among them

Codex's read of c5dec76: the two runs that confirm a kill could each start past the budget with a whole TIMEOUT,
and outlast the step. Each run is now cut to what is left before the budget's end and one TIMEOUT, and one past it
is over budget. With its test and two mutants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: a484f99dc8

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
…s made of an annotation never evaluated

Codex's read of a484f99:
- an import that hangs takes its whole 60 s, so neither the head's probe nor an edit's begins past the budget;
- a fallback mutant Python reads as the head, as of a deferred annotation, is never made.
Each with its test and its mutants.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: ad01122c95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
don-petry and others added 2 commits November 8, 2026 02:54
…e endings, and calls a hung mutant an ERROR

Codex's read of ad01122:
- reading the whole suite imports every test module under a TIMEOUT of its own, so it does not begin past the budget;
- the head's file is read as its bytes, so a CRLF file's edit is not STALE;
- a large hunk any mutant of which hung, or would not run, is an ERROR, never killed by its neighbours.
Each with its test and its mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…rt, and an edit's import probe

The mutant shards at 6f36c4a: "a run of the tests begins past the budget" survived, since the bound at the budget's
end also refused the run; and "an edit's import is probed past the budget" survived, since a spent budget skips the
head's probe and so every edit's. Each now has a test that sees it alone.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 95998a6a95

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
don-petry and others added 2 commits November 8, 2026 03:18
… UNCHECKED, never killed

Codex's read of 95998a6: one mutant killed and five left by the budget made the hunk killed. It is now UNCHECKED,
"over budget", unless a mutant survived, was flaky or hung. With its case and its mutant.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The Factory Builder gave the record 0166: #628 took 0164 and #632 took 0165. Its filename, its title and the
mention in tools/AGENTS.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry don-petry changed the title The advisory check of a changed line of Python no test notices: untested.yml and tools/untested.py (record NNNN) The advisory check of a changed line of Python no test notices: untested.yml and tools/untested.py (record 0166) Nov 8, 2026

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: 24d7e6890e

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
Comment thread tools/untested.py Resolved
don-petry and others added 3 commits November 8, 2026 08:23
…ng name, and never counts a trailer's span

Codex's read of 24d7e68:
- a line is ended by a newline alone, so a form feed or U+2028 in a string no longer moves a revert onto the wrong
  line and makes a real change look the same;
- a crash is read from unittest's whole output, since the mutant run's failures cut a long test's name before its
  exception;
- a trailer is checked against the hunks' lines without making a set of every line in its span.
Each with its test, its mutant and its Boundaries row.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… that now reads it

The trailer's span is now read without a set of its lines, so "a trailer that names no changed line stands" went
STALE. It mutates the new condition.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

…eback mutant's text

Shard 1 at cfb6b30: "the traceback's first line is read as the error" went STALE, since the new mutant's text held
its target a second time in tools/mutants.py. The new one's string is split as the old one's is.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@sonarqubecloud

sonarqubecloud Bot commented Nov 8, 2026

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Nov 8, 2026
Merged via the queue into main with commit 60316bb Nov 8, 2026
24 of 25 checks passed
@don-petry
don-petry deleted the untested-advisory branch November 8, 2026 17:58
don-petry added a commit that referenced this pull request Nov 8, 2026
…ER; 0167 counts eleven

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Nov 9, 2026
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant