Skip to content

fix: tighten rate-limit detector, add is_cli_error, route CLI errors to exit 1 - #149

Merged
don-petry merged 10 commits into
mainfrom
claude/issue-148-20260512-1802
May 13, 2026
Merged

don-petry merged 10 commits into
mainfrom
claude/issue-148-20260512-1802

Conversation

@don-petry

@don-petry don-petry commented May 12, 2026 •

Copy link
Copy Markdown
Collaborator

Summary

  • is_rate_limited hardened: removes over-broad exhausted pattern (matched "retry attempts exhausted" and OS errors); adds missing overloaded keyword (Claude 529 overload response).
  • New is_cli_error function in scripts/engine.sh: detects CLI invocation errors (bad flags, wrong syntax, "command not found") which must never trigger engine fallback.
  • All three call sites in review-one-pr.sh updated to check is_cli_error before is_rate_limited: CLI errors now exit 1 (per-PR failure) instead of exit 2 (session abort).
  • Single-review false-positive bug fixed: the rate-limit check at the single-review tier now requires $single_rc -ne 0, preventing reviewed PR content that mentions rate-limit keywords from aborting the session.
  • Improved logging: [error] CLI format error — treating as per-PR failure (not a rate limit) makes post-mortem analysis unambiguous.
  • Bats tests added (tests/test_rate_limit.bats, tests/test_cli_error_routing.bats): unit tests for all AC-specified inputs/outputs plus an integration routing test that covers the exact Copilot incident from run docs(AGENTS): record dev-lead/pr-review automation rules (hands-off label; pr-review is CODEOWNER approver) #715.

Test plan

  • bats tests/test_rate_limit.bats — all unit tests pass (is_rate_limited false for CLI errors, true for rate-limit strings)
  • bats tests/test_cli_error_routing.bats — routing integration tests pass
  • Verify overloaded_error / 529 / rate limit exceeded / quota exceeded / usage limit reached still exit 2
  • Verify error: Invalid command format. / Did you mean: / unknown flag / command not found now exit 1 (not 2)
  • Confirm a successful model run that mentions "rate limit" in review text does NOT exit 2

Closes #148

Generated with Claude Code

Summary by CodeRabbit

  • Bug Fixes

    • Improved error classification to distinguish CLI invocation/syntax errors from rate-limit/overload conditions, preventing incorrect engine fallbacks and ensuring retries are routed as per-PR failures, engine fallback, or pass-through more reliably.
    • Triage and retry flows now reduce false positives when rate-limit keywords appear in PR content.
  • Tests

    • Added integration suites validating CLI vs. rate-limit routing and boolean checks for error classification.

Review Change Stack

…xit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
@coderabbitai

coderabbitai Bot commented May 12, 2026 •

Copy link
Copy Markdown

No actionable comments were generated in the recent review. 🎉

ℹ️ Recent review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 2f30caf4-1220-43f5-aab4-a777c6f38476

📥 Commits

Reviewing files that changed from the base of the PR and between 5fe7ad0 and 54b3b82.

📒 Files selected for processing (2)
  • scripts/engine.sh
  • tests/test_cli_error_routing.bats

📝 Walkthrough

Walkthrough

This PR narrows rate-limit detection in is_rate_limited(), adds is_cli_error() to classify CLI invocation/syntax failures, and uses both in triage, single-review, and deep-review paths in review-one-pr.sh; unit and integration Bats tests validate routing to exit 1 (per-PR), exit 2 (engine fallback), or pass-through.

Changes

Error Classification System for CLI vs Rate-Limit Failures

Layer / File(s) Summary
Error detection helpers
scripts/engine.sh
is_rate_limited() removes overly-broad exhausted match and adds specific token/overload/billing/HTTP variants; new is_cli_error() matches CLI syntax/flag/unknown-command errors.
Error classification in review flow
scripts/review-one-pr.sh
Triage, single-review retry loop, and deep-review now capture stderr as needed and classify failures: CLI invocation errors exit 1, rate-limit/overload exit 2, otherwise continue.
Unit tests for error detection
tests/test_rate_limit.bats
Asserts is_rate_limited() and is_cli_error() behavior across CLI-format errors and rate-limit/quota patterns, including regression guard for "retry attempts exhausted".
Integration tests for error routing
tests/test_cli_error_routing.bats
Integration scenarios assert routing: "Invalid command format" -> per-PR failure (exit 1); rate-limit/overload/quota/HTTP 529 -> engine-fallback (exit 2); rc=0 remains pass-through; non-matching non-zero errors pass through.

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~25 minutes

Possibly related issues

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The PR title 'fix: tighten rate-limit detector, add is_cli_error, route CLI errors to exit 1' accurately reflects the main changes: hardening is_rate_limited, introducing is_cli_error, and routing CLI errors to exit code 1 instead of 2.
Linked Issues check ✅ Passed The PR successfully addresses all acceptance criteria from #148: it audits and tightens rate-limit patterns [#148 criterion 1-2], adds is_cli_error to distinguish CLI errors routed to exit 1 [#148 criterion 3], improves logging for error categorization [#148 criterion 4], preserves detection of genuine rate limits [#148 criterion 5], and includes unit and integration Bats tests [#148 expected tests].
Out of Scope Changes check ✅ Passed All changes are directly scoped to #148: engine.sh and review-one-pr.sh modifications implement the detector tightening and CLI error routing, and two new test files validate the changes without introducing unrelated functionality.
Docstring Coverage ✅ Passed Docstring coverage is 100.00% which is sufficient. The required threshold is 80.00%.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch claude/issue-148-20260512-1802

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

…minology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
@don-petry

Copy link
Copy Markdown
Collaborator Author

@petry-projects/org-leads — this PR is ready for review and merge.

It fixes #148 (rate-limit detector too broad). Changes are scoped to scripts/engine.sh, scripts/review-one-pr.sh, and new bats tests in tests/. All acceptance criteria from the issue are addressed.

@gemini-code-assist gemini-code-assist Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Code Review

This pull request refines error handling by distinguishing between CLI invocation errors and rate-limit/overload conditions. It adds an is_cli_error function and updates is_rate_limited in scripts/engine.sh to reduce false positives by removing overly broad patterns. The logic in scripts/review-one-pr.sh now correctly routes CLI errors to a per-PR failure (exit 1) and rate limits to an engine fallback (exit 2). Comprehensive tests were added to ensure correct routing. A review comment suggests refactoring the duplicated error routing logic into a shared helper function to improve maintainability.

Comment thread scripts/review-one-pr.sh Resolved

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@scripts/engine.sh`:
- Line 95: Replace the single 429-character inline regex with a maintainable
multi-line pattern list and use grep's -f or process-substitution so each
provider/pattern is on its own line; specifically, extract the long string into
a here-doc (e.g., PATTERNS=$(cat <<'PATTERNS' ... PATTERNS) or use <(cat
<<'PATTERNS' ... PATTERNS)) containing one pattern per line with optional inline
comments, then call grep -qiE -f <(cat <<'PATTERNS' ...) "$input" (or grep -qiE
-f "$PATTERNS_FILE" "$input") instead of the single quoted alternation; update
the location where the original string literal is used to reference this
multi-line source so individual patterns (e.g., those matching "claude", "429",
"quota exceeded") are easy to read and maintain.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: ASSERTIVE

Plan: Pro

Run ID: 52b06323-a16e-49fb-a26f-09b84cd1b140

📥 Commits

Reviewing files that changed from the base of the PR and between 22e46ea and 43a558a.

📒 Files selected for processing (4)
  • scripts/engine.sh
  • scripts/review-one-pr.sh
  • tests/test_cli_error_routing.bats
  • tests/test_rate_limit.bats

Comment thread scripts/engine.sh Resolved

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR hardens the review-session error classification so CLI invocation failures (bad flags/syntax/missing command) no longer get misclassified as provider rate limits that trigger engine fallback, and it fixes a false-positive rate-limit abort in the single-review tier.

Changes:

  • Tightens is_rate_limited patterns (removes bare exhausted, adds overloaded) and adds a new is_cli_error classifier in scripts/engine.sh.
  • Updates scripts/review-one-pr.sh to route CLI errors to exit 1 (per-PR failure) and only route real rate limits to exit 2 (engine fallback), including the single-review $single_rc -ne 0 guard.
  • Adds Bats unit/integration tests covering rate-limit vs CLI-error classification and routing behavior.

Reviewed changes

Copilot reviewed 4 out of 4 changed files in this pull request and generated 3 comments.

File Description
scripts/engine.sh Updates rate-limit detection regex and introduces is_cli_error for CLI invocation error classification.
scripts/review-one-pr.sh Routes failures based on is_cli_error vs is_rate_limited, and fixes single-review false positives by requiring non-zero rc.
tests/test_rate_limit.bats Adds unit tests for is_rate_limited and is_cli_error.
tests/test_cli_error_routing.bats Adds an integration-style routing test for exit-code behavior on CLI errors vs rate limits.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Comment thread tests/test_cli_error_routing.bats Resolved
Comment thread scripts/engine.sh Resolved
Comment thread scripts/engine.sh Resolved
@donpetry-bot

Copy link
Copy Markdown
Contributor

Review — fix requested (cycle 1/3)

The automated review identified the following issues. Please address each one:

Findings to fix

[Findings would be inserted here]

Additional tasks

  1. Resolve all unresolved review thread comments from other reviewers
  2. Ensure all CI checks pass after your changes
  3. Rebase on the target branch if behind
  4. Do NOT modify files unrelated to the findings above

The review cascade will automatically re-review after new commits are pushed.

github-actions Bot and others added 8 commits May 12, 2026 22:38
- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
@sonarqubecloud

Copy link
Copy Markdown

don-petry added a commit that referenced this pull request Jun 7, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 7, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 12, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 14, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 18, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 25, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
don-petry added a commit that referenced this pull request Jun 25, 2026
…to exit 1 (#149)

* fix: tighten is_rate_limited, add is_cli_error, route CLI errors to exit 1

Closes #148

The Copilot CLI (and similar tools) can exit non-zero with a CLI format
error like "error: Invalid command format." which has nothing to do with
API rate limiting.  Previously the single-review call site had no rc guard,
so any non-zero exit would reach is_rate_limited and — if the output ever
matched a pattern — abort the entire batch session with exit 2 instead of
treating it as a per-PR failure.

Changes:
- scripts/engine.sh
  - is_rate_limited: remove standalone `exhausted` (too broad — matched
    "retry attempts exhausted" and similar process output); retain the
    specific `token.*exhaust` / `out of.*token` forms.
  - is_rate_limited: add `overloaded` pattern (Claude 529 overload error).
  - is_cli_error: new function — returns true for bad-flag / wrong-syntax /
    command-not-found output that must NOT be treated as a rate limit.

- scripts/review-one-pr.sh (all three call sites: triage, single-review,
  deep review)
  - Check is_cli_error *before* is_rate_limited at every site; exit 1
    (per-PR failure) on a CLI error so the session continues.
  - Wrap the single-review rate-limit check in `[ "$single_rc" -ne 0 ]`
    to prevent false positives from reviewed PR content that mentions
    rate-limit keywords in its code/comments.
  - Improve log messages: "[error] CLI format error — treating as
    per-PR failure (not a rate limit)" for unambiguous post-mortem analysis.

- tests/test_rate_limit.bats: bats unit tests — is_rate_limited returns
  false for all four CLI error strings from the AC; returns true for all
  five rate-limit strings (overloaded, 529, rate limit, quota, usage limit).
  Also verifies bare "exhausted" no longer causes a false positive.

- tests/test_cli_error_routing.bats: integration test — verifies the
  error-routing helper produces exit 1 for the exact Copilot incident
  (rc=1 + "Invalid command format"), exit 2 for real rate limits, and
  pass-through (0) when rc=0 regardless of PR content.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: narrow 'overloaded' pattern to avoid false positives on code terminology

Replace bare `overloaded` with `overloaded_error|service.*overload|overload.*error`
so that code discussions mentioning "overloaded operator" or "overloaded methods"
in a PR diff do not trigger a spurious rate-limit exit 2.

Also add tests verifying the false-positive guard, and a comment in the
integration test helper clarifying the stdout-only behaviour of the deep
review tier.

Co-authored-by: Don Petry <don-petry@users.noreply.github.com>

* fix: address PR review comments — split regex, printf, test comment

- Split is_rate_limited's single-line regex into grouped segments with
  inline category comments (addresses CodeRabbit maintainability request)
- Replace echo with printf '%s\n' in both is_rate_limited and is_cli_error
  to prevent reinterpretation of leading -n/-e flags (Copilot correctness)
- Narrow test_cli_error_routing.bats header comment to accurately state
  which tiers the integration helper covers (Copilot accuracy request)

All 37 bats tests pass.

Co-Authored-By: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>

---------

Co-authored-by: claude[bot] <41898282+claude[bot]@users.noreply.github.com>
Co-authored-by: Don Petry <don-petry@users.noreply.github.com>
Co-authored-by: github-actions[bot] <41898282+github-actions[bot]@users.noreply.github.com>
Co-authored-by: Gemini CLI <gemini-cli@example.com>
Co-authored-by: Claude Sonnet 4.6 (1M context) <noreply@anthropic.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

fix: rate-limit detector too broad — misclassifies CLI format errors as rate limits

3 participants