Skip to content

feat: implement issue #920 — Skill eval: API throttling masquerades as a held-out regression (infra failure scored as regression, not error) - #925

Merged
don-petry merged 3 commits into
mainfrom
dev-lead/issue-920-20260625-0249
Jun 25, 2026
Merged

don-petry merged 3 commits into
mainfrom
dev-lead/issue-920-20260625-0249

Conversation

@don-petry

Copy link
Copy Markdown
Collaborator

Closes #920

Implemented by dev-lead agent. Please review.

…s a held-out regression (infra failure scored as `regression`, not `error`)
@don-petry
don-petry requested a review from a team as a code owner June 25, 2026 03:03
@chatgpt-codex-connector

Copy link
Copy Markdown

You have reached your Codex usage limits for code reviews. You can see your limits in the Codex usage dashboard.
To continue using code reviews, you can upgrade your account or add credits to your account and enable them for code reviews in your settings.

@coderabbitai

coderabbitai Bot commented Jun 25, 2026 •

Copy link
Copy Markdown

Warning

Review limit reached

@don-petry, we couldn't start this review because you've reached your PR review rate limit.

More reviews will be available in 22 minutes and 29 seconds. Learn how PR review limits work.

Your organization has used up its prepaid credits, and credit purchases are no longer available. Enable the review add-on in the billing tab to keep reviews running — you're only billed for reviews past your plan's rate limits ($0.25/file).

⌛ How to resolve this issue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based credits.

🚦 How do rate limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please see our Fair Usage Limits Policy for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 4154a546-c994-4669-993c-47aaf0e6c78f

📥 Commits

Reviewing files that changed from the base of the PR and between 601dd19 and 738b590.

📒 Files selected for processing (4)
  • scripts/evals/notify-eval-health.sh
  • scripts/evals/run-eval.sh
  • tests/test_eval_health_notify.bats
  • tests/test_skill_evals.bats
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch dev-lead/issue-920-20260625-0249

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry
don-petry enabled auto-merge (squash) June 25, 2026 03:04
@donpetry-bot

Copy link
Copy Markdown
Contributor

Advisory bots were rate-limited; auto-approval is withheld until they recover. pr-review-sweep will re-review this PR after 2026-06-25T04:05:21Z.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request introduces a clear separation between infrastructure failures (such as API throttling) and quality regressions in the evaluation pipeline. It updates run-eval.sh to capture the engine's exit status and exit with code 2 when failures are purely infrastructure-related, and modifies notify-eval-health.sh to manage a separate eval-infra tracking issue. The review feedback focuses on improving script robustness under set -u by validating positional parameters across several functions, adding pre-flight checks for CLI dependencies (jq and gh), and implementing error handling for mktemp calls.

Comment thread scripts/evals/notify-eval-health.sh Resolved
Comment thread scripts/evals/notify-eval-health.sh Resolved
Comment thread scripts/evals/notify-eval-health.sh Resolved
Comment thread scripts/evals/notify-eval-health.sh Resolved
Comment thread scripts/evals/run-eval.sh Resolved
Comment thread scripts/evals/run-eval.sh Resolved
@don-petry
don-petry disabled auto-merge June 25, 2026 03:07
coderabbitai[bot]
coderabbitai Bot previously approved these changes Jun 25, 2026
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — fix-reviews (applied)

Changes committed and pushed.

@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry
don-petry enabled auto-merge (squash) June 25, 2026 03:14
@sonarqubecloud

Copy link
Copy Markdown

@don-petry
don-petry disabled auto-merge June 25, 2026 03:23
@don-petry

Copy link
Copy Markdown
Collaborator Author

Dev-Lead — review-changes (no-changes)

No changes were needed for this PR.

@don-petry
don-petry enabled auto-merge (squash) June 25, 2026 03:26

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: 738b5908363a575205ceadc4f5dd9e4b17894ae8
Review mode: triage-approved (single reviewer)

Summary

Implements #920: the skill-eval scorer now captures each engine call's exit status (engine_rc) so an all-infra-failure run (every model throttled) is classified as outcome=error (exit 2) rather than a held-out regression (exit 1). notify-eval-health.sh splits the single tracker into a regression tracker (eval-health) and a separate infra-error tracker (eval-infra), so a transient throttle can never open/update a false regression issue. Changes are report-only (Phase 1), non-blocking, and confined to scripts/evals/ plus bats tests — no held-out case data touched.

Linked issue analysis

Closes #920. All acceptance criteria are met:

  • All-throttled run (every engine call exits non-zero) → exit 2 / outcome=error, and opens the eval-infra tracker, never the eval-health regression tracker. Covered by test "all-throttled run ... exit 2" and "error never edits or closes an open held-out REGRESSION tracker".
  • Genuine quality miss (engine exits 0, output wrong/unparseable) → exit 1 / regression, unchanged. Covered by "a genuine quality miss ... stays a regression" and "unparseable output with a clean engine exit is a regression".
  • Mixed run (some answered, some throttled) → deterministically exit 2, documented in the run-eval.sh header. Covered by "mixed run ... exit 2".
  • New bats coverage added in tests/test_skill_evals.bats and tests/test_eval_health_notify.bats using stub engine injection (no network).
  • No changes under evals/** — only scripts/ and tests/.

Findings

No blocking findings.

  • Prior advisory feedback from gemini-code-assist (positional-param validation under set -u in score_deterministic/score_llm_judge/find_existing; command -v jq/gh pre-flight checks; mktemp error handling) is fully incorporated in the reviewed head SHA.
  • SonarCloud quality gate passed (0 new issues, 0 security hotspots). CodeQL (actions + python) clean. gitleaks secret scan passed.
  • coderabbit and codex advisory bots were rate-limited (no findings produced); their absence is non-blocking and other reviewers covered the change.
  • Secret-scanning MCP tool was not available in this environment; relied on the green gitleaks CI check instead. The diff is shell logic + tests with no credential material.
  • Design note (non-blocking): a mixed run where some cases pass and the rest throttle exits 2 (infra error), discarding the passing results as "un-scored". This is a deliberate, documented conservative choice — correct for this report-only phase.

CI status

All required checks green: Lint, ShellCheck, shellcheck, bats, unit-tests, validate-agent-profiles, gh-aw-compile, Compile agentic workflows, CodeQL (actions + python), Secret scan (gitleaks), agent-shield/AgentShield, holdout-guard, Test-Deletion Guard, SonarCloud, dev-lead. dependency-audit ecosystems SKIPPED (no matching manifests). mergeStateStatus=BLOCKED only because human review (org-leads) is still required.


Reviewed automatically by the PR-review agent (single-reviewer mode: fable 5). Reply if you need a human review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Skill eval: API throttling masquerades as a held-out regression (infra failure scored as regression, not error)

2 participants