Repository navigation
Evals: the review Intent's held-out set has its 44 cases, and the cutover's rule is held (#183) - #363
Conversation
…over's rule is held (#183) evals/pr-review/cases.json names 44 cases by id and sha256. The cases are in the private home and nowhere here. The rule that an Intent family is cut over only after its eval set exists stops waiting: its test loses @pending, and its rows in evals/AGENTS.md and decision 0010 lose "not held yet". The test of a set of no case now writes its own empty manifest instead of reading this repo's, which is no longer empty. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
|
Warning Review limit reachedYou've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Next included review available in 22 minutes. View limit details
No actionable comments were generated in the recent review.
|
| Layer / File(s) | Summary |
|---|---|
Populate and validate the eval set evals/pr-review/cases.json, tests/test_evals.py, docs/decisions/0010-built-greenfield-cut-over-one-family-at-a-time.md, evals/AGENTS.md |
The manifest now lists 44 case IDs and SHA-256 values. The held-out eval-set test is no longer pending. The empty-set test writes an empty manifest, checks for eval_set_empty without scratch files, then restores the synthetic cases. Two references to the test drop “not held yet.” |
Priority:
Estimated code review effort: 2 (Simple) | ~10 minutes
Change: Other · Severity of issue fixed: Medium
Possibly related issues
- Issue 309 requires the eval workflow to run every case in
evals/pr-review. This PR adds the case manifest but not the workflow. - Issue 194 requires the Harness to use the review Intent. This PR supplies the eval-set manifest but does not implement the Harness.
Possibly related PRs
- combsmith/software-factory#340: This PR adds the case manifest used by the release cut and channel-move eval-set checks.
- combsmith/software-factory#323: This PR populates the eval-set manifest for the
review-changeIntent. - combsmith/software-factory#354: This PR adds the case hashes used by the release eval gate.
Merge Risk: 
Minimal · previous review 7282f
-
No concrete merge-blocking issue is established. The changes appear mergeable subject to normal checks.
Pre-merge checks: 
3 · 
2
-
Failed checks: 2 warnings Passed checks: 3 passed-
Full details: Title check
-
Explanation
The title names the eval work (44 review Intent held-out cases) and the cutover rule being held, which matches the changes to the eval test, the manifest, and the decision/doc rows. However, the title is dense and uses the non-standard phrasing "the cutover's rule is held", and it includes a trailing issue reference. It is specific enough for a teammate to identify the primary change, so it passes.
-
Full details: Linked Issues check
-
Explanation
Issue
#183requires the eval-set cutover references in decision 0010, decision 0013, and the last row ofdocs/context-engineering.md. This PR updates decision 0010, butdocs/context-engineering.mdstill marks the fresh-agent eval asnot held yet. The PR diff also does not update decision 0013. The manifest contains only IDs and hashes, which supports the held-out-content rule. The private home and its case provenance are not independently verifiable from the repository.Resolution
Update the required row in
docs/context-engineering.mdand the corresponding row in decision 0013 to reference the held test or issue#8. Runtools/check_context.pyand include its passing result.
Full details: Docstring Coverage
-
Explanation
Docstring coverage is 0.00% which is insufficient. The required threshold is 80.00%. Docstring coverage is scoped to functions touched by this diff. Documentation-only edits and deletions within surviving functions are included; unchanged diff context and fully deleted functions are excluded. Analyzed 1 functions across 1 files. (3 skipped: 3 unsupported.)
Resolution
Write docstrings for the functions missing them to satisfy the coverage threshold.
-
Finishing Touches: 💡 1
-
📝 Generate docstrings: 💡
-
- Commit to this branch
- Create a new PR
🧪 Generate unit tests (beta)
-
- Commit to this branch
- Create a new PR
-
- Autofix · Keep fixing CodeRabbit findings and required CI, and resolving merge conflicts
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.
Comment @coderabbitai help to get the list of available commands.
…83 asks Twenty cases with a Finding and as many of a change that was right: the test's floor was twenty, which a set of one kind alone would pass. The manifest keeps no case's kind, so the total is what can be held here. From the Claude review of #363. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
…and combsmith#380, folded in (combsmith#387) * The second-family reads of combsmith#341, combsmith#356, combsmith#363 and combsmith#380, folded in Argon read the four after they merged. What it found that holds: - combsmith#380: an approval in the second of the reopening, or of no time, is not recovered; an approval sent on a pull request closed under the act is failed, never refused. Each is tested at its bound and has its mutant. - combsmith#341: an id of True is no account's, though True == 1, in the permission read's typing and in what was seen. Each has its case and its mutant. - combsmith#363: the cutover's rule names its own gate as waiting on release/cutover.py; the test holds the set alone. Not taken: the three guards on a thread's author in adapters/threads.py are equivalent, since the opener is read first by Accounts.actor, which refuses the same; an unvalidated run record is the caller's to hold to its schema, as reconcile's contract says; the manifest's count is held by test_the_manifest_counts_its_cases_and_holds_no_case. Muse's Lens-actor gap on combsmith#356 was closed in its own review round. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * The permission read types an account only for an id that is an int, never True or 1.0 Muse on combsmith#387: an answer of id 1.0 typed the account of id 1, as 1.0 == 1, which the bool guard alone did not refuse. The id must now be an int exactly; a case and a mutant for each of True and 1.0. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>



Closes #183. The last Done-when of "The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests": the cases exist, and the rule that waited on them is held.
What it holds
evals/pr-review/cases.json:count44, and each case's id and the sha256 of its file in the private home. The ids are random and listed sorted, so neither an id nor its place says what a case expects. Nothing else of any case is here. The set meets The review Intent's held-out eval set, from the proof of concept's merged-then-reverted pull requests #183's floor of 20 cases of each kind.test_the_review_intents_eval_set_has_its_casesloses@pending, which it would otherwise fail as an unexpected success. Its rows inevals/AGENTS.mdand decision 0010 lose "not held yet".test_a_set_of_no_case_ends_the_run_before_anything_is_readwrites its own empty manifest instead of reading this repo's. It read the real manifest and asserted it was empty, which no longer holds. Ruling 8's behaviour is tested unchanged.How the home was checked
eval-case/1. Each is under the port's 1 MiB limit.passwithtools/eval_result.pygiven a Result that answers it rightly, andfailgiven one that answers it wrongly. None scoresunreadable.GET /repos/{owner}/{repo}/contents/pr-review/{id}.json, the pathadapters/heldout.pyreads, and hashed to the sha256 here, 44 of 44.Gates
python3 -m tools.verify: 1275 tests OK (12 expected failures), exit 0.maingives the same at a3299a3.make verifyrun on this change exited 1. Its output was cut off, and the full rerun above passed, so the cause is unknown. Possibly a wall-clock test under load (not confirmed).🤖 Generated with Claude Code
Summary by CodeRabbit