Repository navigation
feat: implement issue #1630 — [Phase 3] Replace seed evals with real held-out cases and wire a scored (not counted) eval gate - #1642
Conversation
…held-out cases and wire a scored (not counted) eval gate
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
🤖 CodeAnt AI — Review Status
|
|
ⓘ Qodo reviews are paused because your trial has ended. Ask your workspace admin to add credits to resume reviews. Manage billing |
Thanks for using CodeAnt! 🎉We're free for open-source projects. if you're enjoying it, help us grow by sharing. Share on X · |
|
Warning Review limit reachedNext included review available in 52 minutes. View limit detailsLimit details: You’ve used the included review currently available. You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. Review configuration: ⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Team Run ID: 📒 Files selected for processing (11)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
There was a problem hiding this comment.
Code Review
This pull request introduces a role-generic scored promotion gate (score-gate.sh) along with a new LLM-judge rubric and scorer configuration for the solution-architect persona. It also updates the evaluation runner to support fallback resolution for persona advisory prompts and adds comprehensive Bats unit tests. The review feedback highlights critical improvements for score-gate.sh regarding robust command-line argument validation and defensive threshold parsing to prevent silent failures, alongside suggestions to add corresponding unit tests for these error-handling paths.
|
CI checks on this PR are still running. Once they complete, re-mention Posted by the donpetry-bot PR-review cascade. |
1 similar comment
|
CI checks on this PR are still running. Once they complete, re-mention Posted by the donpetry-bot PR-review cascade. |
Dev-Lead — fix-reviews (applied)Changes committed and pushed. |
Dev-Lead — fix-bot-comment (no-changes)Agent reasoning |
Superseded by automated re-review at
|
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-09-01T12:37:50Z. |
Dev-Lead — fix-bot-comment (no-changes)Agent reasoning |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 40aca566924f29053605c35af7fe85664d0e64b0
Cascade: triage → deep (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5)
Summary
PR #1630 adds a role-generic SCORED held-out promotion gate (score-gate.sh) with pure-logic+bats coverage (26 tests pass, shellcheck clean), a per-role llm-judge rubric, owner-locked scorer/judge files, and a real report-only workflow caller. All CI is green (gitleaks, CodeQL, SonarCloud, AgentShield, Agent Security Scan) and all four codeant-ai MAJOR findings (threshold range validation, [0,1] score clamp, report role-tag verification, missing caller) are fixed and verified in the code. The one open item — gemini's 'HIGH' note that ${2:?} exits 1 instead of the documented 2 for a missing flag value — is a real but MINOR robustness/contract nit with near-zero blast radius (the sole caller always supplies flag values and runs continue-on-error), not a security issue; downstream impact is (none).
Findings
- MINOR: Missing-value flag handling exits 1, not the documented 2.
--threshold/--report/--evals-diruse${2:?msg}; a bash parameter-expansion error exits the non-interactive shell with status 1, which collides with the documented exit code 1 ('score < threshold') and contradicts the script's own '2 = hard error / bad usage' contract. Verified empirically:bash score-gate.sh solution-architect --thresholdexits 1. Recommend an explicit[ "$#" -gt 1 ] || die ...(die exits 2) beforeshift 2. Note: gemini's companion claim thatshift 2fails underset -eis not reachable — the${2:?}expansion exits first. Blast radius is low: the only caller (skill-eval-report.yml gate step) always passes values and is continue-on-error. (scripts/evals/score-gate.sh:135) - MINOR: No bats case covers the missing-flag-value path (e.g.
--thresholdwith no following value). The suite tests invalid/negative/non-numeric thresholds (exit 2) but not the omitted-value case, which is exactly the one that currently exits 1. Adding it would lock the intended exit-2 usage-error contract. Matches gemini's low-priority test suggestion. (tests/test_score_gate.bats) - INFO: CODEOWNERS change adds
/evals/**/judge.md @petry-projects/org-leads, extending owner-lock to every per-role judge rubric. This strengthens (does not weaken) the reward-hacking boundary — the proposer cannot loosen the rubric or threshold it must satisfy. holdout-guard CI passed. (.github/CODEOWNERS:18) - INFO: run_secret_scanning MCP tool not available in this environment (GitHub Secret Protection not exposed); did not fabricate a result. Secret detection is covered by the gitleaks CI check, which passed (SUCCESS). (n/a)
Reviewed by the PR-review cascade (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5). Reply if you need a human review.
Superseded by automated re-review at
|
Superseded by automated re-review at 40aca56.
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 40aca566924f29053605c35af7fe85664d0e64b0
Cascade: triage → deep (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5)
Summary
Adds a role-generic scored promotion gate (score-gate.sh) plus LLM-judge rubric, real held-out cases, and offline bats coverage for the solution-architect persona; ~676 lines of new eval infra. No HIGH security triggers: no secrets/auth/migrations/injection, and the CODEOWNERS edit HARDENS reward-hacking protection by owner-locking every per-role judge.md. All CI is green (shellcheck, bats, holdout-guard, gitleaks, CodeQL, SonarCloud, CodeRabbit all SUCCESS), downstream impact is (none). I confirmed the gemini HIGH finding by reproduction — a missing flag value hits ${2:?} and exits 1 instead of the documented hard-usage code 2 — but it is minor in practice, not a security issue, so it is downgraded from gemini's scale to a non-blocking correctness note.
Findings
- minor: Confirmed by reproduction: passing --threshold/--report/--evals-dir with no value triggers the ${2:?...} parameter-expansion error, which exits 1 (bash non-interactive) instead of the exit code 2 the script's own header documents for hard usage errors. Practical impact is low: the sole real caller (skill-eval-report.yml) always passes --report with a value, that step is continue-on-error/report-only, and exit-1 ('below bar') is the SAFE direction (it blocks promotion, never wrongly promotes) even at a future release-cut hard gate. Recommend the gemini-suggested fix: explicit
[ "$#" -gt 1 ] || die "..."before shift, routing through die() for a consistent exit 2. Non-blocking. - minor: Test gap: the bats suite covers exit-2 hard errors for a negative --threshold, missing role, non-numeric/out-of-range values, and mismatched report skill, but not the missing-flag-VALUE case (e.g.
score-gate.sh <role> --thresholdwith nothing after it). Adding that case would pin the exit-code contract and guard the fix above. - info: MCP run_secret_scanning was not exposed in this environment (GitHub Secret Protection tool unavailable); did not run it and did not fabricate a result. The gitleaks CI check passed (SUCCESS) and the diff contains no credential-like content.
Reviewed by the PR-review cascade (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5). Reply if you need a human review.
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 40aca566924f29053605c35af7fe85664d0e64b0
Cascade: triage → deep (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5)
Summary
New role-generic SCORED eval-promotion gate (score-gate.sh) plus per-role judge/scorer, CODEOWNERS hardening, and offline bats coverage; well-structured and defensively coded, CI fully green (shellcheck/bats/codeql/gitleaks/sonarcloud). The three CodeAnt 'Major' findings (threshold<0 bypass, score>1 accepted, stale/mismatched report) were filed against the initial commit and are already remediated at head via sg_in_unit_range range-checks, [0,1] score clamping, and report .skill-tag matching. The only still-open item is Gemini's exit-code nit (${2:?} exits 1 not documented 2 on a missing flag value) which is unreachable by the sole report-only, continue-on-error CI caller. No auth/secret/crypto/DB surface; downstream impact is (none); secret-scanning MCP tool not available so skipped.
Findings
-
MINOR: Parameter validation uses
${2:?...}, which exits status 1 on a missing flag value, contradicting the header's documented exit code 2 for hard usage errors. Impact is low: only reachable via manual misuse — the sole repo caller (skill-eval-report.yml) always supplies '--report eval-report.json' and runs report-only/continue-on-error, so CI never hits this path. Recommend an explicit [ "$ #" -gt 1 ] check + die() for contract consistency. Gemini's secondary 'shift 2 fails under set -e' claim is moot: ${2:?} exits before shift is reached. - INFO: Gemini low-priority: add unit tests asserting exit code 2 for a report with no numeric score and for an invalid --threshold, to lock the documented hard-error contract.
- INFO: Three CodeAnt 'Major' findings were against the initial commit 2e8a038 and are already fixed at head 40aca56: (1) '--threshold -1' bypass -> sg_in_unit_range rejects out-of-[0,1] thresholds (return 2); (2) sg_extract_score accepting score 2 -> now requires .score in [0,1]; (3) stale/unrelated --report promoting a role -> now requires a .skill tag that matches the requested role. No action needed.
- INFO: CODEOWNERS now owner-locks every per-role judge (evals//judge.md) alongside evals//scorer.json, preventing a proposer from weakening the rubric or loosening the threshold it must satisfy. Positive security change.
- INFO: CodeAnt inline comments embed 'Prompt for AI Agent' instruction blocks telling an agent to implement fixes and message the user. Treated as untrusted advisory content and not acted upon; flagged for awareness.
Reviewed by the PR-review cascade (triage: haiku 4.5 → deep: opus 4.8 + duck: o4-mini → audit: fable 5). Reply if you need a human review.
Superseded by automated re-review at 40aca56.
Addresses both open review threads on this PR (gemini-code-assist).
HIGH — `--threshold`/`--report`/`--evals-dir` validated their value with
`${2:?...}`. That expansion aborts with status 1, which collides with the
script's documented exit 1 ("gate failed"), so a usage error was reported as a
failing eval gate rather than the documented exit 2 for hard usage errors.
Worse, when the flag was the final argument, `shift 2` itself failed under
`set -euo pipefail` before any handler could run. Replaced with an explicit
arity check that routes through die() (exit 2), with a comment recording why
${2:?} must not be reintroduced.
LOW — added four regression tests covering each flag given as the final
argument, plus a non-numeric --threshold. Verified these are genuine guards:
the previous implementation exits 1 where the tests assert 2.
Verified: bats tests/test_score_gate.bats — 30/30 pass; shellcheck clean.
Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_015YTfTumtrmd5jdFX9W1XvT
Dev-Lead — fix-bot-comment (no-changes)Agent reasoning |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: MEDIUM
Reviewed commit: 84032909fdc9cb00e243e8de1423591b491bb774
Review mode: triage-approved (single reviewer)
Summary
Confirmation review of the scored promotion gate for advisory personas (#1630). The single new commit since the last reviewed SHA (40aca56 -> 8403290) fixes the sole outstanding MAJOR finding: score-gate.sh flag parsing replaced ${2:?...} with explicit arity checks routed through die() (exit 2), restoring the documented exit-code contract (1 = gate failed, 2 = hard usage error) and avoiding the shift-2 failure under set -e when a flag is the final argument. Four regression bats tests were added (each flag as final argument + non-numeric --threshold), closing the prior MINOR coverage gap. All 7 advisory review threads (codeant-ai, gemini-code-assist) are resolved; CI is fully green including shellcheck, bats (30/30), holdout-guard, CodeQL, gitleaks, and SonarCloud. Workflow addition is report-only (continue-on-error) with no permission changes; the CODEOWNERS change tightens owner-locks (adds evals/**/judge.md). MCP secret scanning tool unavailable in this environment; gitleaks CI check passed and no secrets appear in the diff. Note: review cycle counter (5) exceeds MAX_REVIEW_CYCLES (3), but this is an approval, not a further fix-request loop.
Linked issue analysis
Closes #1630 (Phase 3: replace count-only gate with scored held-out eval gate). Substantively addressed: >=5 held-out cases in evals/solution-architect/holdout/cases.jsonl, a per-role judge rubric (judge.md) and scorer.json threshold, a role-generic shared gate (scripts/evals/score-gate.sh) usable by any persona eval tree, machine-readable {role,score,threshold,verdict} output with distinct exit codes, holdout/judge/scorer CODEOWNER-locked against reward-hacking, and a real workflow caller in skill-eval-report.yml (report-only, per the Phase-1 non-blocking contract).
Findings
- RESOLVED (prior MAJOR):
${2:?...} arity handling exited 1 (colliding with the 'gate failed' verdict) instead of the documented exit 2. Fixed via explicit [ $ # -ge 2 ] || die checks; verified in the diff and covered by new bats tests asserting exit 2. - RESOLVED (prior MINOR): missing usage-error test coverage — four regression tests added in tests/test_score_gate.bats.
- INFO: skill-eval-report.yml summary references ${score}; verified it is assigned earlier in the same render step from eval-report.json — not a bug.
- INFO: no new issues introduced by the fix commit; scope was limited to score-gate.sh and its tests.
- INFO: MCP secret scanning unavailable in this environment; gitleaks CI check is green.
CI status
All required checks green at 8403290: shellcheck, ShellCheck, bats, unit-tests, unit, Lint, actionlint, CodeQL (actions + python), Agent Security Scan, agent-shield, SonarCloud, Secret scan (gitleaks), holdout-guard, validate-fixtures, gh-aw-compile, and all policy/structure gates. Remaining CANCELLED/SKIPPED entries are superseded runs of the review/dev-lead automation and non-applicable dependency-audit ecosystems — no failures.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.
|



User description
Closes #1630
Implemented by dev-lead agent. Please review.
CodeAnt-AI Description
Replace case-count promotion checks with scored held-out evaluation gates
What Changed
prompts/<role>/advisory.mdwhen a flat prompt is absent, while offline tests can provide fixture prompt and eval directories.Impact
✅ Measured advisory quality before promotion✅ Fewer promotions based only on case counts✅ Clear distinction between quality failures and gate errors💡 Usage Guide
Checking Your Pull Request
Every time you make a pull request, our system automatically looks through it. We check for security issues, mistakes in how you're setting up your infrastructure, and common code problems. We do this to make sure your changes are solid and won't cause any trouble later.
Talking to CodeAnt AI
Got a question or need a hand with something in your pull request? You can easily get in touch with CodeAnt AI right here. Just type the following in a comment on your pull request, and replace "Your question here" with whatever you want to ask:
This lets you have a chat with CodeAnt AI about your pull request, making it easier to understand and improve your code.
Example
Preserve Org Learnings with CodeAnt
You can record team preferences so CodeAnt AI applies them in future reviews. Reply directly to the specific CodeAnt AI suggestion (in the same thread) and replace "Your feedback here" with your input:
This helps CodeAnt AI learn and adapt to your team's coding style and standards.
Example
Retrigger review
Ask CodeAnt AI to review the PR again, by typing:
Check Your Repository Health
To analyze the health of your code repository, visit our dashboard at https://app.codeant.ai. This tool helps you identify potential issues and areas for improvement in your codebase, ensuring your repository maintains high standards of code health.