Repository navigation
fix: tighten rate-limit detector, add is_cli_error, route CLI errors to exit 1 - #149
Conversation
…xit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
|
No actionable comments were generated in the recent review. 🎉 ℹ️ Recent review info⚙️ Run configurationConfiguration used: Organization UI Review profile: ASSERTIVE Plan: Pro Run ID: 📒 Files selected for processing (2)
📝 WalkthroughWalkthroughThis PR narrows rate-limit detection in ChangesError Classification System for CLI vs Rate-Limit Failures
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
…minology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
|
@petry-projects/org-leads — this PR is ready for review and merge. It fixes #148 (rate-limit detector too broad). Changes are scoped to |
There was a problem hiding this comment.
Code Review
This pull request refines error handling by distinguishing between CLI invocation errors and rate-limit/overload conditions. It adds an is_cli_error function and updates is_rate_limited in scripts/engine.sh to reduce false positives by removing overly broad patterns. The logic in scripts/review-one-pr.sh now correctly routes CLI errors to a per-PR failure (exit 1) and rate limits to an engine fallback (exit 2). Comprehensive tests were added to ensure correct routing. A review comment suggests refactoring the duplicated error routing logic into a shared helper function to improve maintainability.
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@scripts/engine.sh`:
- Line 95: Replace the single 429-character inline regex with a maintainable
multi-line pattern list and use grep's -f or process-substitution so each
provider/pattern is on its own line; specifically, extract the long string into
a here-doc (e.g., PATTERNS=$(cat <<'PATTERNS' ... PATTERNS) or use <(cat
<<'PATTERNS' ... PATTERNS)) containing one pattern per line with optional inline
comments, then call grep -qiE -f <(cat <<'PATTERNS' ...) "$input" (or grep -qiE
-f "$PATTERNS_FILE" "$input") instead of the single quoted alternation; update
the location where the original string literal is used to reference this
multi-line source so individual patterns (e.g., those matching "claude", "429",
"quota exceeded") are easy to read and maintain.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Organization UI
Review profile: ASSERTIVE
Plan: Pro
Run ID: 52b06323-a16e-49fb-a26f-09b84cd1b140
📒 Files selected for processing (4)
scripts/engine.shscripts/review-one-pr.shtests/test_cli_error_routing.batstests/test_rate_limit.bats
There was a problem hiding this comment.
Pull request overview
This PR hardens the review-session error classification so CLI invocation failures (bad flags/syntax/missing command) no longer get misclassified as provider rate limits that trigger engine fallback, and it fixes a false-positive rate-limit abort in the single-review tier.
Changes:
- Tightens
is_rate_limitedpatterns (removes bareexhausted, addsoverloaded) and adds a newis_cli_errorclassifier inscripts/engine.sh. - Updates
scripts/review-one-pr.shto route CLI errors to exit1(per-PR failure) and only route real rate limits to exit2(engine fallback), including the single-review$single_rc -ne 0guard. - Adds Bats unit/integration tests covering rate-limit vs CLI-error classification and routing behavior.
Reviewed changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.
| File | Description |
|---|---|
scripts/engine.sh |
Updates rate-limit detection regex and introduces is_cli_error for CLI invocation error classification. |
scripts/review-one-pr.sh |
Routes failures based on is_cli_error vs is_rate_limited, and fixes single-review false positives by requiring non-zero rc. |
tests/test_rate_limit.bats |
Adds unit tests for is_rate_limited and is_cli_error. |
tests/test_cli_error_routing.bats |
Adds an integration-style routing test for exit-code behavior on CLI errors vs rate limits. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
Review — fix requested (cycle 1/3)The automated review identified the following issues. Please address each one: Findings to fix[Findings would be inserted here] Additional tasks
The review cascade will automatically re-review after new commits are pushed. |
- Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
|
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
…to exit 1 (#149) * fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1 Closes #148 The Copilot CLI (and similar tools) can exit non-zero with a CLI format error like "error: Invalid command format." which has nothing to do with API rate limiting. Previously the single-review call site had no rc guard, so any non-zero exit would reach is_rate_limited and — if the output ever matched a pattern — abort the entire batch session with exit 2 instead of treating it as a per-PR failure. Changes: - scripts/engine.sh - is_rate_limited: remove standalone `exhausted` (too broad — matched "retry attempts exhausted" and similar process output); retain the specific `token.*exhaust` / `out of.*token` forms. - is_rate_limited: add `overloaded` pattern (Claude 529 overload error). - is_cli_error: new function — returns true for bad-flag / wrong-syntax / command-not-found output that must NOT be treated as a rate limit. - scripts/review-one-pr.sh (all three call sites: triage, single-review, deep review) - Check is_cli_error *before* is_rate_limited at every site; exit 1 (per-PR failure) on a CLI error so the session continues. - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]` to prevent false positives from reviewed PR content that mentions rate-limit keywords in its code/comments. - Improve log messages: "[error] CLI format error — treating as per-PR failure (not a rate limit)" for unambiguous post-mortem analysis. - tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns false for all four CLI error strings from the AC; returns true for all five rate-limit strings (overloaded, 529, rate limit, quota, usage limit). Also verifies bare "exhausted" no longer causes a false positive. - tests/test_cli_error_routing.bats: integration test — verifies the error-routing helper produces exit 1 for the exact Copilot incident (rc=1 + "Invalid command format"), exit 2 for real rate limits, and pass-through (0) when rc=0 regardless of PR content. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: narrow 'overloaded' pattern to avoid false positives on code terminology Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error` so that code discussions mentioning "overloaded operator" or "overloaded methods" in a PR diff do not trigger a spurious rate-limit exit 2. Also add tests verifying the false-positive guard, and a comment in the integration test helper clarifying the stdout-only behaviour of the deep review tier. Co-authored-by: Don Petry <don-petry@users.noreply.github.com> * fix: address PR review comments — split regex, printf, test comment - Split is_rate_limited's single-line regex into grouped segments with inline category comments (addresses CodeRabbit maintainability request) - Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error to prevent reinterpretation of leading -n/-e flags (Copilot correctness) - Narrow test_cli_error_routing.bats header comment to accurately state which tiers the integration helper covers (Copilot accuracy request) All 37 bats tests pass. Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com> --------- Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com> Co-authored-by: Don Petry <don-petry@users.noreply.github.com> Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com> Co-authored-by: Gemini CLI <gemini-cli@example.com> Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>



Summary
is_rate_limitedhardened: removes over-broadexhaustedpattern (matched "retry attempts exhausted" and OS errors); adds missingoverloadedkeyword (Claude 529 overload response).is_cli_errorfunction inscripts/engine.sh: detects CLI invocation errors (bad flags, wrong syntax, "command not found") which must never trigger engine fallback.review-one-pr.shupdated to checkis_cli_errorbeforeis_rate_limited: CLI errors now exit 1 (per-PR failure) instead of exit 2 (session abort).$single_rc -ne 0, preventing reviewed PR content that mentions rate-limit keywords from aborting the session.[error] CLI format error — treating as per-PR failure (not a rate limit)makes post-mortem analysis unambiguous.tests/test_rate_limit.bats,tests/test_cli_error_routing.bats): unit tests for all AC-specified inputs/outputs plus an integration routing test that covers the exact Copilot incident from run docs(AGENTS): record dev-lead/pr-review automation rules (hands-off label; pr-review is CODEOWNER approver) #715.Test plan
bats tests/test_rate_limit.bats— all unit tests pass (is_rate_limited false for CLI errors, true for rate-limit strings)bats tests/test_cli_error_routing.bats— routing integration tests passoverloaded_error/529/rate limit exceeded/quota exceeded/usage limit reachedstill exit 2error: Invalid command format./Did you mean:/unknown flag/command not foundnow exit 1 (not 2)Closes #148
Generated with Claude Code
Summary by CodeRabbit
Bug Fixes
Tests