Skip to content

Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported - #159

Merged
don-petry merged 5 commits into
mainfrom
kernel/rubric-coverage
Oct 18, 2026
Merged

don-petry merged 5 commits into
mainfrom
kernel/rubric-coverage

Conversation

@don-petry

@don-petry don-petry commented Oct 18, 2026 •

Copy link
Copy Markdown
Contributor

Closes #16.

A reviewing Run that leaves a criterion of the rubric uncovered has not reported. It doesn't count as the model review a merge needs, and the existing attempts brake bounds the retries. The rulings are on #16, and decision 0029 records them.

What changes

  • kernel/coverage.py (new, protected): one pure function, covered, decides covered, uncovered (with the criterion ids) or unreadable. It judges the rubric's criteria against the Run's coverage list and Findings, and the changed paths, including both names of a rename.
  • coverage/1: a schema of its own for the coverage list, with an example and a counter-example. Schemas for Task, Result, Effect and Disposition #19 is left waiting on its trials.
  • No new fact: coverage lives in the Result, which the Kernel never reads. The run record carries the outcome, conclusion: failure with outcome: item. Only a successful review Run counts as a report. Failed ones are re-admitted, and after 3 attempts the item is held.
  • New reason, run_unreadable: a "success" Run that also carries a fault, from a required Lens's Role, defers to the Maintainer. A queued item leaves the queue.
  • ** in the one path syntax: it now means any number of names, in both kernel/derive.py and tools/hooks/protected.py, and the agreement test covers it. The hooks' check for a directory holding a protected path doesn't handle ** yet, so load() refuses ** in any protected, reported or Lens-input entry until it does.
  • Simulator: uncovered reviews occur on one seed in six.

Found while building, filed separately: #158. "Kernel: spent attempts hold the item even while a refused Run is queued". With attempts spent, a queued Run that the brakes would refuse hides the Hold, so the Factory moves the item from held to verifying, a move the Lifecycle doesn't declare. No Effect and no model spend follow. This predates this story. The simulator's narrower spread, UNCOVERED_SEEDS, carries a comment naming #158, which restores it.

Verified:

  • make verify passes (540 tests) on a plain git archive export with PATH=/usr/bin:/bin. The record was renumbered there only, for the gap; this PR is red in CI only on that gap until 0027 and 0028 merge.
  • Each new mutant was checked by hand against its own test, and is killed by an assertion. No local mutation run; CI's shards run the set.
  • check_decisions is clean, and the simulator passes at 300 seeds.

Reviewed by: a fresh-context Claude review, a security pass, and AGY (Gemini 4 Argon) on c66e1ee. Nothing found lets an uncovered review reach an approval. Their findings are fixed in ed9b014:

  • the criterion-id checks, the required-Role guard and each clause of contradicts had no test or mutant;
  • the dedupe and readability checks were untested;
  • ** and the directory-holds check.

Left, informational:

  • coverage.py is protected but imports the unprotected derive.matches. kernel/ is a Lens input at PR time, so a change to it needs every model Lens's review (0020).
  • A Finding that cites a criterion not in the rubric still gives covered.
  • Until apply calls covered, coverage is enforced by whoever writes the run record. That is the harness row in 0029.

Order: merges after #155 (decision 0028).

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

don-petry and others added 2 commits October 17, 2026 20:30
…not reported

Builds "Kernel: a reviewing Run that leaves a rubric criterion uncovered has
not reported" (issue 16) as its working rulings give it, recorded in decision
0029.

- kernel/coverage.py: covered(rubric, coverage, findings, changed) says
  covered, uncovered with the ids, or unreadable. A protected path.
- schemas/coverage.schema.json (coverage/1), its example, and a
  counter-example valid to the schema that covered refuses;
  tools/check_examples.py runs covered over both.
- The path syntax gains ** as a whole name, any run of names, none included,
  in kernel/derive.py's matches and the hooks' _glob alike; the agreement
  test reads the rubric's patterns too.
- A required model Lens has reported only through a review of this content
  that succeeded. A review that succeeded with an outcome named is
  run_unreadable: a deferral for the Maintainer, naming the Run, and it
  takes a queued item out.
- The simulator: on one seed in six a review is now and then uncovered;
  three invariants (I-37 at approval and merge, a Lens's thread only from a
  review that succeeded, uncovered reviews at most max_attempts).
- Mutants for each rule, each naming the test that states it; three moved
  mutants follow the code they break.
- Design 3.10's item row names an uncovered reviewing Result.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…he hooks cannot read it

The review round on decision 0029, from the Claude review, the security pass
and a second model family.

- kernel/coverage.py: an id with no letters, or no digits after its hyphen,
  is no criterion id and is shown as `unreadable`, never as text from
  outside; an id given twice is named once; a rubric naming a criterion
  twice and paths not read whole are unreadable. Each has a case and a
  mutant.
- kernel/derive.py: a contradictory review Run by a Role no required Lens
  is, the author's among them, or on a dependency update no model Lens
  reviews, is not read; nor one of another Intent, on other content, not
  completed, or that failed. Each clause and the Role guard has a case and
  a mutant.
- tools/hooks/protected.py: load() refuses `**` as a whole name in a
  protected, reported or Lens-input path, since the directory-holds check
  reads each name as one name. Tested, with a mutant.
- sim/world.py: the narrowed spread of uncovered reviews names "Kernel:
  spent attempts hold the item even while a refused Run is queued"
  (issue 158), whose fix widens it again.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 18, 2026 01:34
@coderabbitai

coderabbitai Bot commented Oct 18, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 1 minute.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

⚙️ Run configuration
  • Configuration used: Repository: combsmith/software-factory/.coderabbit.yaml
  • Review profile: CHILL
  • Plan: Advanced
  • Run ID: 1b85b242-b5c3-4b7e-8732-c8abb68d29e9
📥 Commits

Reviewing files that changed from the base of the PR and between 5389cd0 and 0647125.

📒 Files selected for processing: 24
  • README.md
  • config/hooks.json
  • docs/architecture/software-factory-design.md
  • docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md
  • docs/glossary.md
  • examples/coverage.json
  • examples/invalid/coverage.a-criterion-that-applies-left-out.json
  • kernel/AGENTS.md
  • kernel/coverage.py
  • kernel/decide.py
  • kernel/derive.py
  • kernel/terms.py
  • schemas/coverage.schema.json
  • sim/AGENTS.md
  • sim/world.py
  • tests/AGENTS.md
  • tests/test_coverage.py
  • tests/test_decide.py
  • tests/test_hooks.py
  • tests/test_simulator.py
  • tests/test_tools.py
  • tools/check_examples.py
  • tools/hooks/protected.py
  • tools/mutants.py
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@chatgpt-codex-connector

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ⚠️ Failed 2026-10-18T01:34:44.786037Z ed9b014 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@don-petry don-petry left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Code Review against config/rubric-pr-review.json

P1 Badge Major / High finding (PR-5): Renumber decision record to avoid numbering gap

  • Location: docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md
  • Rubric Criterion: PR-5 (Requirements, Major: "A rule that is new, or that has stopped waiting, names a test that would fail if the rule were broken").
  • Finding: The parent branch (origin/main) contains decisions only through 0026. Introducing 0029 leaves a numbering gap at 0027 and 0028.
  • Impact: python3 -m tools.check_context fails with:
    • docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md: records are numbered from 0001 with no gap
    • docs/decisions/0029-a-reviewing-run-that-leaves-a-criterion-uncovered-has-not-reported.md: the title must open with its number
      This failure cascades into tests/test_tools.py::Verify.test_every_check_has_a_test_in_which_it_fails, causing make verify and all CI mutation shards to fail before breaking any mutant (the tests fail before anything is broken; a mutant cannot be told from that).
  • Remedy: Renumber this record to the next consecutive decision number or rebase once PRs #153 (0027) and #155 (0028) merge to main.

@don-petry don-petry left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Code Review against config/rubric-pr-review.json

Outcome: PASS

All criteria evaluated against commit 8717a91 (merged with origin/main at 5389cd0):

  • PR-1 (Requirements, Major): PASS. Delivers Issue #152 ("Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported"). Adds coverage tracking and reporting constraints without out-of-scope additions.
  • PR-2 (Security, Blocking): PASS. No unauthorized file writes or broadened effects.
  • PR-3 (Functional, Major): PASS. Comprehensive test coverage for rubric criteria coverage evaluation, unreadable criteria identification, and missing criteria handling.
  • PR-4 (Architecture, Blocking): PASS. Kernel purity verified; no I/O, network, or clock reads introduced in kernel/.
  • PR-5 (Requirements, Major): PASS. The previous numbering gap finding is resolved. With PR #155 (Decision 0028) merged to main, Decision 0029 is now strictly contiguous. All context checks (python3 -m tools.check_context) pass with no errors and line/character boundaries fully respected.

CI's mutation run on #159 found "a wildcard crosses a separator" alive: no
test could kill it. Since `**`, `fits` is given only a stretch of the path
with as many names as the pattern's run (`reached`), so the separators in
that stretch are the pattern's own and a wildcard can never stand for one.
`within_a_name` could never fail. It goes, with the two mutants that broke
it; the one on a missing piece now breaks `found < 0` alone, and is killed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@don-petry don-petry left a comment

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Adversarial Code Review against config/rubric-pr-review.json

Outcome: PASS

Evaluated commit 0647125:

  • PR-1 (Requirements, Major): PASS. Addresses issue #152 without extraneous changes.
  • PR-2 (Security, Blocking): PASS. No unauthorized file writes or broadened permissions.
  • PR-3 (Functional, Major): PASS. In fits, wildcards cannot cross directory separators because reached() slices paths by directory segments; within_a_name was redundant and its surviving mutants in tools/mutants.py were pruned. The piece mutant now cleanly targets found < 0 and is caught by existing tests.
  • PR-4 (Architecture, Blocking): PASS. Pure kernel invariant intact; no clock reads or I/O introduced.
  • PR-5 (Requirements, Major): PASS. Decision 0029 contiguous; context checks pass cleanly.

@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Oct 18, 2026
Merged via the queue into main with commit af27660 Oct 18, 2026
16 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Kernel: a reviewing Run that leaves a rubric criterion uncovered has not reported

1 participant