Repository navigation
Mutation run: a kill is confirmed, and one copy per worker - #153
Conversation
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 26 minutes. View limit details
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info
📝 Walkthrough
Merge Risk: ⚪ Minimal · up to
|
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
d163dba to
bfd031c
Compare
Adversarial Review (AGY - Gemini 3.8 Flash High)Evaluated against Rubric Evaluation
Findings
Verdict: PASS |
There was a problem hiding this comment.
💡 Codex Review
Here are some automated review suggestions for this pull request.
Reviewed commit: d163dba3cf
ℹ️ About Codex in GitHub
Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you
- Open a pull request for review
- Mark a draft as ready
- Comment "@codex review".
If Codex has suggestions, it will comment; otherwise it will react with 👍.
Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".
Adversarial Review (AGY - Gemini 3.8 Flash High)Evaluated against Rubric Evaluation
Findings
Verdict: PASS |
A failing test counted as a kill, whatever made it fail, so a test that failed under load passed four rules nothing checked as checked. Now the one test that failed runs alone again on the mutant, where it must fail, and alone on the unbroken code, where it must pass. A failure short of that runs the suite once more: a pass is SURVIVED, a confirmed failure is killed, and a second unconfirmed one is FLAKY, which is not counted and fails the run. A timeout on any run stays an ERROR. The test is read from the line unittest prints under its rule of =, in the 3.9/3.10 and 3.11+ forms, a subtest's and setUpClass's among them. The suite is now discovered from the top of the copy, as verify.py does, so a test's id is its dotted path and can be run alone. Tests timed by the wall clock are marked @wall_clock, skipped when mutating and run as they are in make verify; a test holds that each one that reads the clock or sleeps carries the mark. Decision 0027, lesson L-60, and eleven mutants of the runner's own rules. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…or each rule From the review, the security pass and a second model's read. - The unbroken side of a confirmation runs in a second copy, made as the mutant's is. The checkout has a .git and is not under the system's temporary directory; a test that fails only in a copy failed on every mutant and passed unbroken, so a change could make its own mutants look killed. A test holds that such a test kills nothing. - A test named to run alone is run alone in a suite of more than one. - Mutants for the rules that had none: the copy, `only`, a confirmation that errors, a failure line that cannot be read, and a subtest suffix. - The @wall_clock test asserts the marked set equals the timed set, by file, class and method, and reads `from time import` and `import time as` too. - The tools/AGENTS.md row names a test that checks what it says. - The shard time in verify.yml and its two copies. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Every mutant copied the whole tree into a fresh directory, and the last round added a second copy for each kill. Over hundreds of mutants and several runs that wrote hundreds of gigabytes and stopped the machine. Closes #152. - Each worker copies the tree once and is lent to one mutant at a time. A mutant writes only the file it breaks, with a time of its own past the file's so no bytecode compiled from another is taken for it, and however its runs end, a crash or a hang among them, the file's own bytes and time are written back and read again. - The unbroken side of a confirmation runs in the same copy with the file put back: an export on both sides, so .git is absent on both. - A copy whose file does not read back, or that is lent with a file that is not the tree's, is an ERROR for that mutant and never lent again. - At most four workers unless told. - A test holds that a mutant leaves no trace for the next, and mutants hold the restore, the guard, the one copy and the unbroken side. One copy is 2.8 MB, 227 files, with no __pycache__ and no .git. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
b47e1f7 to
0043c68
Compare
Adversarial Review (AGY - Gemini 3.8 Flash High)Evaluated against Rubric Evaluation
Findings
Verdict: PASS |
… confirmation on ERROR
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|



Closes #147. Closes #152.
Why. Two problems with the mutation runner:
What changes
SURVIVED; a second unconfirmed failure isFLAKY, which fails the run (a working ruling).ERROR.@wall_clockand skipped during the mutation run; they still run inmake verify. A test holds that the marked set is exactly the set of tests that read the clock, by file, class and method, includingfrom time importandimport time as..gitand no caches. It runs mutants one at a time in that copy.ERROR, and the copy is never reused.-s tests -t .), so a failing test's id can be run alone. This matchestools/verify.py.tools/AGENTS.mdnames its test, and lesson L-60 is added.verify.ymland its two copies is updated. The figure is an estimate until CI shows it.Verified:
make verifypasses (524 tests) on a plaingit archiveexport withPATH=/usr/bin:/bin; the record was renumbered in the export only, for the 0027 gap..gitcomes outFLAKY.Reviewed by: a fresh-context Claude review, a security pass, and AGY (Gemini 4 Argon) on 9abeffd. All their findings are fixed in 4f31038:
onlycase was untested;AGENTS.mdrow and the time comment.The copy-once rework in d163dba was read again by a focused Claude review and by AGY (Gemini 4 Argon), and AGY approved with no findings. The review found no high or medium bugs. Its three low findings all fail safe:
🤖 Generated with Claude Code
Summary by CodeRabbit