Repository navigation
feat: implement issue #920 — Skill eval: API throttling masquerades as a held-out regression (infra failure scored as regression, not error) - #925
Conversation
…s a held-out regression (infra failure scored as `regression`, not `error`)
|
You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard. |
|
Warning Review limit reached
More reviews will be available in 22 minutes and 29 seconds. Learn how PR review limits work. Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file). ⌛ How to resolve this issue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits. 🚦 How do rate limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please see our Fair Usage Limits Policy for further information. ℹ️ Review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (4)
✨ Finishing Touches🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
|
Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-25T04:05:21Z. |
There was a problem hiding this comment.
Code Review
This pull request introduces a clear separation between infrastructure failures (such as API throttling) and quality regressions in the evaluation pipeline. It updates run-eval.sh to capture the engine's exit status and exit with code 2 when failures are purely infrastructure-related, and modifies notify-eval-health.sh to manage a separate eval-infra tracking issue. The review feedback focuses on improving script robustness under set -u by validating positional parameters across several functions, adding pre-flight checks for CLI dependencies (jq and gh), and implementing error handling for mktemp calls.
Dev-Lead — fix-reviews (applied)Changes committed and pushed. |
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
|
Dev-Lead — review-changes (no-changes)No changes were needed for this PR. |
donpetry-bot
left a comment
There was a problem hiding this comment.
Automated review — APPROVED ✓
Risk: LOW
Reviewed commit: 738b5908363a575205ceadc4f5dd9e4b17894ae8
Review mode: triage-approved (single reviewer)
Summary
Implements #920: the skill-eval scorer now captures each engine call's exit status (engine_rc) so an all-infra-failure run (every model throttled) is classified as outcome=error (exit 2) rather than a held-out regression (exit 1). notify-eval-health.sh splits the single tracker into a regression tracker (eval-health) and a separate infra-error tracker (eval-infra), so a transient throttle can never open/update a false regression issue. Changes are report-only (Phase 1), non-blocking, and confined to scripts/evals/ plus bats tests — no held-out case data touched.
Linked issue analysis
Closes #920. All acceptance criteria are met:
- All-throttled run (every engine call exits non-zero) → exit 2 / outcome=error, and opens the eval-infra tracker, never the eval-health regression tracker. Covered by test "all-throttled run ... exit 2" and "error never edits or closes an open held-out REGRESSION tracker".
- Genuine quality miss (engine exits 0, output wrong/unparseable) → exit 1 / regression, unchanged. Covered by "a genuine quality miss ... stays a regression" and "unparseable output with a clean engine exit is a regression".
- Mixed run (some answered, some throttled) → deterministically exit 2, documented in the run-eval.sh header. Covered by "mixed run ... exit 2".
- New bats coverage added in tests/test_skill_evals.bats and tests/test_eval_health_notify.bats using stub engine injection (no network).
- No changes under evals/** — only scripts/ and tests/.
Findings
No blocking findings.
- Prior advisory feedback from gemini-code-assist (positional-param validation under
set -uin score_deterministic/score_llm_judge/find_existing;command -v jq/ghpre-flight checks; mktemp error handling) is fully incorporated in the reviewed head SHA. - SonarCloud quality gate passed (0 new issues, 0 security hotspots). CodeQL (actions + python) clean. gitleaks secret scan passed.
- coderabbit and codex advisory bots were rate-limited (no findings produced); their absence is non-blocking and other reviewers covered the change.
- Secret-scanning MCP tool was not available in this environment; relied on the green gitleaks CI check instead. The diff is shell logic + tests with no credential material.
- Design note (non-blocking): a mixed run where some cases pass and the rest throttle exits 2 (infra error), discarding the passing results as "un-scored". This is a deliberate, documented conservative choice — correct for this report-only phase.
CI status
All required checks green: Lint, ShellCheck, shellcheck, bats, unit-tests, validate-agent-profiles, gh-aw-compile, Compile agentic workflows, CodeQL (actions + python), Secret scan (gitleaks), agent-shield/AgentShield, holdout-guard, Test-Deletion Guard, SonarCloud, dev-lead. dependency-audit ecosystems SKIPPED (no matching manifests). mergeStateStatus=BLOCKED only because human review (org-leads) is still required.
Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.



Closes #920
Implemented by dev-lead agent. Please review.