Skip to content

fix(pr-review): detect provider usage-cap and fall back to next engine - #107

Merged
don-petry merged 4 commits into
mainfrom
fix/provider-limit-fallback
May 11, 2026
Merged

don-petry merged 4 commits into
mainfrom
fix/provider-limit-fallback

Conversation

@don-petry

Copy link
Copy Markdown
Collaborator

Problem

Closes #106

When Claude hits its subscription/token usage cap, claude --print exits with code 1 and writes a usage-cap message to stdout. The is_rate_limited function only matched API-level rate-limit phrasing (429, too many requests, etc.) — it missed Claude Code's billing/plan-cap messages entirely.

Result: the cascade hard-failed (exit 1) and aborted the session, instead of exiting 2 and triggering the existing gemini → copilot fallback chain in review-batch.sh.

Observed failure: https://github.com/petry-projects/.github-private/actions/runs/25648497571/job/75282153268


Changes

scripts/engine.sh — extend is_rate_limited regex

Added three new failure-mode categories to the pattern:

Category New patterns
Subscription/billing cap out of.*token, token.*exhaust, claude.*usage, usage.*claude, plan.*limit, subscription.*limit, billing.*limit, daily.*limit, monthly.*limit
HTTP payment-required 402
Service overload (hard block) 529

scripts/review-one-pr.sh — dual-channel rate-limit detection + diagnostics

  1. Check stderr too: $TRIAGE_LOG (stderr) is now read into $TRIAGE_STDERR and passed through is_rate_limited alongside $TRIAGE_RESULT (stdout). Some providers write their limit message to stderr.

  2. Surface the error on hard-fail: the previous failure path only printed stderr via cat $TRIAGE_LOG, silently discarding stdout — where Claude Code actually writes its error. Both channels are now printed so the failure reason is always visible in CI logs.


Behaviour after this fix

Scenario Before After
Claude usage-cap hit at triage Hard-fail, session aborted exit 2 → retry with Gemini or Copilot
Triage fails for other reason Opaque log stdout + stderr both printed
API-level rate limit (already worked) exit 2 → fallback unchanged ✓

Copilot AI review requested due to automatic review settings May 11, 2026 03:29
@coderabbitai

coderabbitai Bot commented May 11, 2026 •

Copy link
Copy Markdown

Warning

Rate limit exceeded

@don-petry has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 58 seconds before requesting another review.

You’ve run out of usage credits. Purchase more in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2b4800bc-785b-47c8-8295-be4de0be60a7

📥 Commits

Reviewing files that changed from the base of the PR and between 6bce841 and b5eedbd.

📒 Files selected for processing (2)
  • scripts/engine.sh
  • scripts/post-pr-review.sh
✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/provider-limit-fallback

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request enhances the rate-limit and usage-cap detection logic by expanding the regex patterns in is_rate_limited to cover subscription limits, billing caps, and specific Claude usage messages. It also updates the triage process to check both stdout and stderr for these patterns and improves error logging. A review comment correctly identifies a risk of false positives due to the broader regex patterns and suggests guarding the detection logic with a check for a non-zero exit code to ensure it only triggers when the command actually fails.

Comment thread scripts/review-one-pr.sh Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Improves the PR-review engine cascade by detecting provider “usage cap”/billing-limit style failures (not just classic API rate limits) so the workflow can exit with code 2 and fall back to the next engine instead of hard-failing the session.

Changes:

  • Expand is_rate_limited matching to include subscription/billing-cap phrasing plus HTTP 402 and 529 signals.
  • In triage, detect limit messages from both stdout and stderr, and print both channels when triage hard-fails for better CI diagnostics.

Reviewed changes

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

File Description
scripts/review-one-pr.sh Adds dual-channel (stdout+stderr) limit detection and prints both streams on triage hard-fail.
scripts/engine.sh Broadens is_rate_limited regex to detect usage-cap/billing-limit/service-overload blocks.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread scripts/review-one-pr.sh Resolved
coderabbitai[bot]
coderabbitai Bot previously approved these changes May 11, 2026

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

Copilot reviewed 2 out of 2 changed files in this pull request and generated 1 comment.

Comment thread scripts/engine.sh Resolved
GitHub Copilot and others added 3 commits May 10, 2026 23:05
When Claude Code hits a subscription/token usage cap it exits with
code 1 and writes a usage-limit message to stdout. The previous
is_rate_limited regex only covered API-level rate-limit phrasing (429,
"too many requests", etc.) and missed Claude Code's billing/plan-cap
messages — so the cascade hard-failed instead of falling back through
the existing gemini → copilot chain in review-batch.sh.

Changes:
- engine.sh: extend is_rate_limited to cover subscription/billing caps
  (out of tokens, claude.*usage, plan/billing/subscription limit, HTTP
  402 payment-required, 529 overloaded).
- review-one-pr.sh: check $TRIAGE_LOG (stderr) as well as $TRIAGE_RESULT
  (stdout) — some providers write their limit message to stderr.
- review-one-pr.sh: surface both stdout and stderr in the hard-fail path
  so the actual error message is always visible in CI logs.

Fixes #106

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Remove claude.*usage / usage.*claude (already covered by 'usage limit'
substring match) and daily/monthly.*limit (too broad, false-positive
risk on PR diffs). Condense 6-line comment to 3 lines.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
PR description listed these patterns but they were absent from the regex.
Adds claude.*usage, usage.*claude, daily.*limit, monthly.*limit to ensure
Claude Code billing-cap messages are actually matched.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@don-petry
don-petry force-pushed the fix/provider-limit-fallback branch from dac63ef to 80d184d Compare May 11, 2026 04:06
coderabbitai[bot]
coderabbitai Bot previously approved these changes May 11, 2026
The single-review prompt documents 'skip' as a valid return value for
draft PRs and head-SHA mismatches. post-pr-review.sh only handled
'approve' and 'escalate', causing the review session to hard-fail
(exit 1) rather than gracefully no-op (exit 100) when the reviewer
returned skip.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

@donpetry-bot donpetry-bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Automated review — APPROVED ✓

Risk: LOW
Reviewed commit: b5eedbdca7a116e1f4bb37174277468170464497
Review mode: triage-approved (single reviewer)

Summary

Extends is_rate_limited() in engine.sh to detect provider subscription/billing caps (plan limits, out-of-tokens, HTTP 402/529) so the engine fallback chain fires instead of hard-failing the session. Also adds a missing skip decision handler in post-pr-review.sh (exit 100 for draft PRs / SHA mismatches). Both changes are additive, safe, and well-scoped.

Linked issue analysis

Closes #106. The primary root cause (incomplete is_rate_limited regex missing Claude Code billing-cap phrasing) is directly addressed by the regex expansion. The skip handler fixes a secondary issue discovered during the same investigation.

Note: The PR description references changes to review-one-pr.sh (dual-channel stderr detection, improved diagnostic output) that are not present in the final diff — those changes were in intermediate commits but appear to have been reverted. Root causes #2 and #3 from the issue (stderr not checked, stdout swallowed on hard-fail) remain open. Consider a follow-up if those enhancements are still desired.

Findings

  • No security concerns. Changes are limited to regex pattern matching and a conditional branch — no auth, secrets, eval, or injection surface.
  • Regex patterns are reasonable. The new patterns (out of.*token, plan.*limit, billing.*limit, HTTP 402/529, etc.) are specific enough to avoid false positives in the rate-limit detection context, especially since the function is called on provider output text after a command failure.
  • skip handler is correct. Placed before the existing unknown-decision validation, extracts the reason from JSON, and exits with the expected code 100.
  • All prior review comments resolved. Gemini and Copilot both flagged the false-positive risk from broad patterns on successful triage output — the author confirmed this was gated on non-zero exit codes (though that specific change isn't in the final diff since review-one-pr.sh changes were reverted).

CI status

Check Status
CodeQL (actions) ✅ SUCCESS
SonarCloud ✅ SUCCESS (0 new issues, 0 security hotspots)
AgentShield ✅ SUCCESS
Dependency audit ✅ SUCCESS/SKIPPED (no applicable ecosystems)
CodeRabbit ✅ SUCCESS

All checks passed.


Reviewed automatically by the PR-review agent (single-reviewer). Reply if you need a human review.

@donpetry-bot
donpetry-bot enabled auto-merge (squash) May 11, 2026 04:22
@don-petry

Copy link
Copy Markdown
Collaborator Author

@claude[agent] - update base branch

@don-petry
don-petry merged commit 28a5cb1 into main May 11, 2026
15 checks passed
@don-petry
don-petry deleted the fix/provider-limit-fallback branch May 11, 2026 04:23
Copilot stopped work on behalf of don-petry due to an error May 11, 2026 04:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix(pr-review): provider usage-cap not detected — cascade hard-fails instead of falling back

3 participants