Repository navigation
Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported - #159
Conversation
…not reported Builds "Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported" (issue 16) as its working rulings give it, recorded in decision 0029. - kernel/coverage.py: covered(rubric, coverage, findings, changed) says covered, uncovered with the ids, or unreadable. A protected path. - schemas/coverage.schema.json (coverage/1), its example, and a counter-example valid to the schema that covered refuses; tools/check_examples.py runs covered over both. - The path syntax gains ** as a whole name, any run of names, none included, in kernel/derive.py's matches and the hooks' _glob alike; the agreement test reads the rubric's patterns too. - A required model Lens has reported only through a review of this content that succeeded. A review that succeeded with an outcome named is run_unreadable: a deferral for the Maintainer, naming the Run, and it takes a queued item out. - The simulator: on one seed in six a review is now and then uncovered; three invariants (I-37 at approval and merge, a Lens's thread only from a review that succeeded, uncovered reviews at most max_attempts). - Mutants for each rule, each naming the test that states it; three moved mutants follow the code they break. - Design 3.10's item row names an uncovered reviewing Result. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he hooks cannot read it The review round on decision 0029, from the Claude review, the security pass and a second model family. - kernel/coverage.py: an id with no letters, or no digits after its hyphen, is no criterion id and is shown as `unreadable`, never as text from outside; an id given twice is named once; a rubric naming a criterion twice and paths not read whole are unreadable. Each has a case and a mutant. - kernel/derive.py: a contradictory review Run by a Role no required Lens is, the author's among them, or on a dependency update no model Lens reviews, is not read; nor one of another Intent, on other content, not completed, or that failed. Each clause and the Role guard has a case and a mutant. - tools/hooks/protected.py: load() refuses `**` as a whole name in a protected, reported or Lens-input path, since the directory-holds check reads each name as one name. Tested, with a mutant. - sim/world.py: the narrowed spread of uncovered reviews names "Kernel: spent attempts hold the item even while a refused Run is queued" (issue 158), whose fix widens it again. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 1 minute. View limit details
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
don-petry
left a comment
There was a problem hiding this comment.
Adversarial Code Review against config/rubric-pr-review.json
Major / High finding (PR-5): Renumber decision record to avoid numbering gap
- Location:
docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md - Rubric Criterion: PR-5 (Requirements, Major: "A rule that is new, or that has stopped waiting, names a test that would fail if the rule were broken").
- Finding: The parent branch (
origin/main) contains decisions only through0026. Introducing0029leaves a numbering gap at0027and0028. - Impact:
python3 -m tools.check_contextfails with:docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md: records are numbered from 0001 with no gapdocs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md: the title must open with its number
This failure cascades intotests/test_tools.py::Verify.test_every_check_has_a_test_in_which_it_fails, causingmake verifyand all CI mutation shards to fail before breaking any mutant (the tests fail before anything is broken; a mutant cannot be told from that).
- Remedy: Renumber this record to the next consecutive decision number or rebase once PRs #153 (
0027) and #155 (0028) merge tomain.
# Conflicts: # kernel/AGENTS.md # tests/AGENTS.md # tools/mutants.py
don-petry
left a comment
There was a problem hiding this comment.
Adversarial Code Review against config/rubric-pr-review.json
Outcome: PASS
All criteria evaluated against commit 8717a91 (merged with origin/main at 5389cd0):
- PR-1 (Requirements, Major): PASS. Delivers Issue #152 ("Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported"). Adds coverage tracking and reporting constraints without out-of-scope additions.
- PR-2 (Security, Blocking): PASS. No unauthorized file writes or broadened effects.
- PR-3 (Functional, Major): PASS. Comprehensive test coverage for rubric criteria coverage evaluation, unreadable criteria identification, and missing criteria handling.
- PR-4 (Architecture, Blocking): PASS. Kernel purity verified; no I/O, network, or clock reads introduced in
kernel/. - PR-5 (Requirements, Major): PASS. The previous numbering gap finding is resolved. With PR #155 (Decision 0028) merged to
main, Decision 0029 is now strictly contiguous. All context checks (python3 -m tools.check_context) pass with no errors and line/character boundaries fully respected.
CI's mutation run on #159 found "a wildcard crosses a separator" alive: no test could kill it. Since `**`, `fits` is given only a stretch of the path with as many names as the pattern's run (`reached`), so the separators in that stretch are the pattern's own and a wildcard can never stand for one. `within_a_name` could never fail. It goes, with the two mutants that broke it; the one on a missing piece now breaks `found < 0` alone, and is killed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
don-petry
left a comment
There was a problem hiding this comment.
Adversarial Code Review against config/rubric-pr-review.json
Outcome: PASS
Evaluated commit 0647125:
- PR-1 (Requirements, Major): PASS. Addresses issue #152 without extraneous changes.
- PR-2 (Security, Blocking): PASS. No unauthorized file writes or broadened permissions.
- PR-3 (Functional, Major): PASS. In
fits, wildcards cannot cross directory separators becausereached()slices paths by directory segments;within_a_namewas redundant and its surviving mutants intools/mutants.pywere pruned. The piece mutant now cleanly targetsfound < 0and is caught by existing tests. - PR-4 (Architecture, Blocking): PASS. Pure kernel invariant intact; no clock reads or I/O introduced.
- PR-5 (Requirements, Major): PASS. Decision 0029 contiguous; context checks pass cleanly.
|



Closes #16.
A reviewing Run that leaves a criterion of the rubric uncovered has not reported. It doesn't count as the model review a merge needs, and the existing attempts brake bounds the retries. The rulings are on #16, and decision 0029 records them.
What changes
kernel/coverage.py(new, protected): one pure function,covered, decides covered, uncovered (with the criterion ids) or unreadable. It judges the rubric's criteria against the Run's coverage list and Findings, and the changed paths, including both names of a rename.coverage/1: a schema of its own for the coverage list, with an example and a counter-example. Schemas for Task, Result, Effect and Disposition #19 is left waiting on its trials.conclusion: failurewithoutcome: item. Only a successful review Run counts as a report. Failed ones are re-admitted, and after 3 attempts the item is held.run_unreadable: a "success" Run that also carries a fault, from a required Lens's Role, defers to the Maintainer. A queued item leaves the queue.**in the one path syntax: it now means any number of names, in bothkernel/derive.pyandtools/hooks/protected.py, and the agreement test covers it. The hooks' check for a directory holding a protected path doesn't handle**yet, soload()refuses**in any protected, reported or Lens-input entry until it does.Found while building, filed separately: #158. "Kernel: spent attempts hold the item even while a refused Run is queued". With attempts spent, a queued Run that the brakes would refuse hides the Hold, so the Factory moves the item from held to verifying, a move the Lifecycle doesn't declare. No Effect and no model spend follow. This predates this story. The simulator's narrower spread,
UNCOVERED_SEEDS, carries a comment naming #158, which restores it.Verified:
make verifypasses (540 tests) on a plaingit archiveexport withPATH=/usr/bin:/bin. The record was renumbered there only, for the gap; this PR is red in CI only on that gap until 0027 and 0028 merge.check_decisionsis clean, and the simulator passes at 300 seeds.Reviewed by: a fresh-context Claude review, a security pass, and AGY (Gemini 4 Argon) on c66e1ee. Nothing found lets an uncovered review reach an approval. Their findings are fixed in ed9b014:
contradictshad no test or mutant;**and the directory-holds check.Left, informational:
coverage.pyis protected but imports the unprotectedderive.matches.kernel/is a Lens input at PR time, so a change to it needs every model Lens's review (0020).covered.applycallscovered, coverage is enforced by whoever writes the run record. That is the harness row in 0029.Order: merges after #155 (decision 0028).
🤖 Generated with Claude Code