Skip to content

Evals: the review Intent's held-out set has its 44 cases, and the cutover's rule is held (#183) - #363

Merged
don-petry merged 2 commits into
mainfrom
evals/183-review-intent-cases
Oct 28, 2026
Merged

don-petry merged 2 commits into
mainfrom
evals/183-review-intent-cases

Conversation

@don-petry

@don-petry don-petry commented Oct 27, 2026 •

Copy link
Copy Markdown
Contributor

Closes #183. The last Done-when of "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests": the cases exist, and the rule that waited on them is held.

What it holds

  • evals/pr-review/cases.json: count 44, and each case's id and the sha256 of its file in the private home. The ids are random and listed sorted, so neither an id nor its place says what a case expects. Nothing else of any case is here. The set meets The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests #183's floor of 20 cases of each kind.
  • The cutover's rule stops waiting. test_the_review_intents_eval_set_has_its_cases loses @pending, which it would otherwise fail as an unexpected success. Its rows in evals/AGENTS.md and decision 0010 lose "not held yet".
  • test_a_set_of_no_case_ends_the_run_before_anything_is_read writes its own empty manifest instead of reading this repo's. It read the real manifest and asserted it was empty, which no longer holds. Ruling 8's behaviour is tested unchanged.

How the home was checked

  • Each of the 44 files validates as eval-case/1. Each is under the port's 1 MiB limit.
  • Each scores pass with tools/eval_result.py given a Result that answers it rightly, and fail given one that answers it wrongly. None scores unreadable.
  • Each was read back through GET /repos/{owner}/{repo}/contents/pr-review/{id}.json, the path adapters/heldout.py reads, and hashed to the sha256 here, 44 of 44.
  • How the cases were chosen is written in the home's README, not here.

Gates

  • python3 -m tools.verify: 1275 tests OK (12 expected failures), exit 0. main gives the same at a3299a3.
  • An earlier make verify run on this change exited 1. Its output was cut off, and the full rerun above passed, so the cause is unknown. Possibly a wall-clock test under load (not confirmed).

🤖 Generated with Claude Code

View guided diff Turn on auto-fix

Summary by CodeRabbit

  • Documentation
    • Clarified the evaluation-set guidance in the project’s decision record and review instructions.
  • Tests
    • Added 44 review evaluation cases.
    • Added validation that an empty evaluation set is reported and does not create temporary files.

…over's rule is held (#183)

evals/pr-review/cases.json names 44 cases by id and sha256. The cases are in the private home and nowhere here.
The rule that an Intent family is cut over only after its eval set exists stops waiting: its test loses @pending,
and its rows in evals/AGENTS.md and decision 0010 lose "not held yet". The test of a set of no case now writes
its own empty manifest instead of reading this repo's, which is no longer empty.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@don-petry
don-petry requested a review from a team as a code owner October 27, 2026 23:56
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review ✅ Completed 2026-10-28T00:38:05.681765Z 3eb9bde New commits
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@coderabbitai

coderabbitai Bot commented Oct 27, 2026 •

Copy link
Copy Markdown

Review in Change Stack →

Warning

Review limit reached

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

Next included review available in 22 minutes.

Check out review usage here.

View limit details

Limit details: You’ve used the included review currently available.

Learn how review limits work.

Review configuration:

Run configuration
Commits

Reviewing files that changed from the base of the PR and between 7282f05 and 3eb9bde.

Files selected for processing: 1
  • tests/test_evals.py

No actionable comments were generated in the recent review.

Recent review info
Run configuration
Commits

Reviewing files that changed from the base of the PR and between a3299a3 and 7282f05.

Files selected for processing: 4
  • docs/decisions/0010-built-greenfield-cut-over-one-family-at-a-time.md
  • evals/AGENTS.md
  • evals/pr-review/cases.json
  • tests/test_evals.py

Included review availability: This review used your included allowance. Your plan provides up to 1 included review per hour; 0 remain after this review.

Walkthrough

The PR adds 44 case IDs and SHA-256 values to the review eval-set manifest. It enables the held-out eval-set test and updates the empty-set test to use an explicit empty manifest. Two eval-set references no longer say “not held yet.”

Changes: Review eval set

Layer / File(s) Summary
Populate and validate the eval set
evals/pr-review/cases.json, tests/test_evals.py, docs/decisions/0010-built-greenfield-cut-over-one-family-at-a-time.md, evals/AGENTS.md
The manifest now lists 44 case IDs and SHA-256 values. The held-out eval-set test is no longer pending. The empty-set test writes an empty manifest, checks for eval_set_empty without scratch files, then restores the synthetic cases. Two references to the test drop “not held yet.”

Priority: Normal

Estimated code review effort: 2 (Simple) | ~10 minutes

Change: Other · Severity of issue fixed: Medium

Possibly related issues

  • Issue 309 requires the eval workflow to run every case in evals/pr-review. This PR adds the case manifest but not the workflow.
  • Issue 194 requires the Harness to use the review Intent. This PR supplies the eval-set manifest but does not implement the Harness.

Possibly related PRs

Merge Risk: Minimal · previous review 7282f

No concrete merge-blocking issue is established. The changes appear mergeable subject to normal checks.

Pre-merge checks: Passed 3 · Failed 2

Failed Failed checks: 2 warnings

Check name Status Explanation Resolution
Linked Issues check Warning Issue #183 requires the eval-set cutover references in decision 0010, decision 0013, and the last row of docs/context-engineering.md. This PR updates decision 0010, but docs/context-engineering.md… Update the required row in docs/context-engineering.md and the corresponding row in decision 0013 to reference the held test or issue #8. Run tools/check_context.py and include its passing result.
Docstring Coverage Warning Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within survi… Write docstrings for the functions missing them to satisfy the coverage threshold.
Passed Passed checks: 3 passed
Check name Status Explanation
Description Check Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check Passed The title names the eval work (44 review Intent held-out cases) and the cutover rule being held, which matches the changes to the eval test, the manifest, and the decision/doc rows. However, the title…
Out of Scope Changes check Passed The changed manifest, eval tests, decision 0010 row, and eval documentation all support issue #183's held-out eval set and cutover requirements. No unrelated change is shown.
Full details: Title check

Explanation

The title names the eval work (44 review Intent held-out cases) and the cutover rule being held, which matches the changes to the eval test, the manifest, and the decision/doc rows. However, the title is dense and uses the non-standard phrasing "the cutover's rule is held", and it includes a trailing issue reference. It is specific enough for a teammate to identify the primary change, so it passes.

Full details: Linked Issues check

Explanation

Issue #183 requires the eval-set cutover references in decision 0010, decision 0013, and the last row of docs/context-engineering.md. This PR updates decision 0010, but docs/context-engineering.md still marks the fresh-agent eval as not held yet. The PR diff also does not update decision 0013. The manifest contains only IDs and hashes, which supports the held-out-content rule. The private home and its case provenance are not independently verifiable from the repository.

Resolution

Update the required row in docs/context-engineering.md and the corresponding row in decision 0013 to reference the held test or issue #8. Run tools/check_context.py and include its passing result.

Full details: Docstring Coverage

Explanation

Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files. (3 skipped: 3 unsupported.)

Resolution

Write docstrings for the functions missing them to satisfy the coverage threshold.

Finishing Touches: 💡 1
📝 Generate docstrings: 💡
  • Commit to this branch
  • Create a new PR
🧪 Generate unit tests (beta)
  • Commit to this branch
  • Create a new PR
  • Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

…83 asks

Twenty cases with a Finding and as many of a change that was right: the test's floor was
twenty, which a set of one kind alone would pass. The manifest keeps no case's kind, so the
total is what can be held here. From the Claude review of #363.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry added this pull request to the merge queue Oct 28, 2026
Merged via the queue into main with commit d556992 Oct 28, 2026
19 checks passed
@don-petry
don-petry deleted the evals/183-review-intent-cases branch October 28, 2026 01:12
YasirBinMudassir pushed a commit to YasirBinMudassir/software-factory that referenced this pull request Nov 7, 2026
…and combsmith#380, folded in (combsmith#387)

* The second-family reads of combsmith#341, combsmith#356, combsmith#363 and combsmith#380, folded in

Argon read the four after they merged. What it found that holds:
- combsmith#380: an approval in the second of the reopening, or of no time, is not recovered; an approval sent on a pull
  request closed under the act is failed, never refused. Each is tested at its bound and has its mutant.
- combsmith#341: an id of True is no account's, though True == 1, in the permission read's typing and in what was seen.
  Each has its case and its mutant.
- combsmith#363: the cutover's rule names its own gate as waiting on release/cutover.py; the test holds the set alone.

Not taken: the three guards on a thread's author in adapters/threads.py are equivalent, since the opener is read
first by Accounts.actor, which refuses the same; an unvalidated run record is the caller's to hold to its schema,
as reconcile's contract says; the manifest's count is held by test_the_manifest_counts_its_cases_and_holds_no_case.
Muse's Lens-actor gap on combsmith#356 was closed in its own review round.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

* The permission read types an account only for an id that is an int, never True or 1.0

Muse on combsmith#387: an answer of id 1.0 typed the account of id 1, as 1.0 == 1, which the bool guard alone did not refuse.
The id must now be an int exactly; a case and a mutant for each of True and 1.0.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

---------

Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests

1 participant