Skip to content

pr-review never auto-approves when advisory bots are rate-limited/out-of-quota — advisory-gate blocks on bots that can't submit #657

Closed

Description

@don-petry

Summary

The pr-review agent is not failing — it is skipping auto-approval on green PRs with decision: skip, reason: waiting-for-advisory-bots, and the skip never clears. Result: clean, CI-green, rebased PRs sit at REVIEW_REQUIRED indefinitely because donpetry-bot never posts its org-leads approval.

Root cause: the advisory-review-gate waits for advisory bots to submit a review at head, but two of the four advisory bots are out of quota and cannot submit, and the gate does not degrade gracefully when a bot is rate-limited.

Evidence

pr-review run for PR #623 (review/review job 81223083427, run 27478909009):

[advisory-gate] Advisory bots detected: gemini-code-assist sonarqubecloud
[advisory-gate] WARNING: Only 2/4 advisory bots submitted so far (head age: ~60s, last submission: ~Ns ago)
[advisory-gate] WARNING: Will re-check when remaining bots submit their reviews
skip: advisory bots still reviewing (non-blocking, will re-check on bot submission)
{"pr":".../623","decision":"skip","reason":"waiting-for-advisory-bots"}

The two missing/non-responding advisory bots are out of quota — confirmed by their own comments across #479, #483, #516, #518, #520, #525, #550, #580, #621, #622, #623:

  • CodeRabbit: "Review limit reached … Your organization has used up its prepaid credits, and credit purchases are no longer available."
  • chatgpt-codex-connector: "You have reached your Codex usage limits for code reviews."

Root cause (code)

scripts/lib/advisory-review-gate.sh:

  • ADVISORY_BOTS (L32–37) = gemini-code-assist, copilot-pull-request-reviewer, sonarqubecloud, chatgpt-codex-connector.
  • check_advisory_reviews (L128–235) requires num_submitted == total_advisory_bots (all 4) or one of two timeout fallbacks: head-commit age > 1200s (L222) or quiescence > 600s (L224). Otherwise return 1 → pr-review skips (review-one-pr.sh L140–177, reason=waiting-for-advisory-bots).
  • In practice the gate rarely sees 4/4: Codex is permanently out of quota, and Copilot reviews only a subset of PRs. So approval depends entirely on the timeout fallbacks — which only re-evaluate on an event re-trigger (a bot submission) or the hourly stuck-review sweep. Every dev-lead push (e.g. a rebase) resets the head-age clock to 0, so freshly-rebased PRs restart the 20-min wait and keep skipping.
  • A RATE_LIMITED display state already exists (format_bot_status L110) but nothing ever sets it — get_advisory_bot_states (L59–85) only maps a bot's literal review/comment state. So the half-built rate-limit handling never triggers.

Impact

This is the gate behind the current backlog of green-CI PRs stuck at REVIEW_REQUIRED (e.g. #479, #483, #516, #518, #520, #525, #550, #589, #620, #621, #622, #623). With CodeRabbit + Codex credits exhausted, the advisory reviews won't arrive, so the gate only clears via the 20-min/10-min fallbacks after a quiet period + a re-trigger — slow and fragile, and reset by any new push.

Recommended fix (pick, keep minimal + tests)

  1. Wire up RATE_LIMITED (preferred). In get_advisory_bot_states, detect a bot comment whose body matches the known rate-limit/usage-limit markers (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits") and emit state: RATE_LIMITED. Treat RATE_LIMITED (and UNSUPPORTED) bots as responded/non-participating so they don't hold the gate. This makes the gate degrade gracefully the moment a bot signals it's out.
  2. Shorten / decouple the fallback so a bounded wait reliably proceeds even without a re-trigger (pair with the sweep), and/or anchor head-age so a no-op rebase doesn't reset the clock.
  3. Config escape hatch: allow dropping a bot from the required advisory set (keep it advisory-only) via env, so an out-of-quota bot (Codex) can be removed without code change.

Separate (infra/billing — not code)

CodeRabbit and Codex have exhausted prepaid credits org-wide, so they aren't reviewing at all right now. Even with a perfect gate, restoring those credits (or accepting their absence via fix #1/#3) is needed for advisory coverage.


Filed from a triage of why pr-review wasn't approving green PRs. Evidence: run 27478909009 (#623) + rate-limit comments on the PRs listed above.

Activity

  1. don-petry commented on Jun 13, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #657 — pr-review never auto-approves when advisory bots are rate-limited/out-of-quota

    Scope

    Implement recommended fix #1 (preferred): wire up the half-built RATE_LIMITED state in scripts/lib/advisory-review-gate.sh so an out-of-quota advisory bot that posts a usage-limit comment is detected, classified RATE_LIMITED, and treated as non-participating — dropping it from the required advisory set so it can no longer hold the gate open indefinitely.

    Implementation Checklist

    • scripts/lib/advisory-review-gate.sh — add known rate-limit/usage-limit marker regex (CodeRabbit "Review limit reached"/"used up its prepaid credits", Codex "reached your Codex usage limits")
    • scripts/lib/advisory-review-gate.sh — get_advisory_bot_states: emit state: RATE_LIMITED for an advisory-bot comment whose body matches a marker
    • scripts/lib/advisory-review-gate.sh — check_advisory_reviews: count RATE_LIMITED/UNSUPPORTED bots as responded and subtract them from the required total (effective_total, clamped to ≥1) so a permanently out-of-quota bot doesn't block approval
    • Keep timeout fallbacks unchanged (head-age / quiescence)

    Tests to Write (TDD, in tests/dev-lead/unit/test_advisory_review_gate.bats)

    • Runtime: rate-limited Codex comment + 2 real reviews (copilot absent), recent head → gate returns 0 (regression: previously skipped/return 1)
    • Classification: a Codex usage-limit comment is rendered as RATE_LIMITED in gate output
    • Negative: a plain Codex comment (no marker) is NOT excluded — gate still waits (return 1) when a genuine bot is absent
    • Structural: script references RATE_LIMITED detection + effective_total

    Test Command

    bats tests/dev-lead/unit/ (full repo lint: bash .dev-lead/scripts/dev-lead-lint.sh)

  2. don-petry commented on Jun 13, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Implemented recommended fix #1 (preferred) — wired up the half-built RATE_LIMITED state so an out-of-quota advisory bot can no longer hold the gate open.

    Plan Execution

    • Added RATE_LIMIT_MARKERS regex (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits")
    • get_advisory_bot_states: a comment from an advisory bot whose body matches a marker is now classified state: RATE_LIMITED (case-insensitive) instead of COMMENTED
    • check_advisory_reviews: RATE_LIMITED/UNSUPPORTED bots are treated as responded and subtracted from the required total (effective_total = total − unavailable, clamped to ≥1); the approval threshold now compares against effective_total
    • Timeout fallbacks (head-age / quiescence) left unchanged; fallback log lines reworded to report the effective required count
    • TDD: 5 new bats tests written before implementation

    Behaviour

    With Codex out of quota (posts its usage-limit comment) and Copilot absent, two genuine advisory reviews (Gemini + SonarCloud) now satisfy the gate immediately — effective_total drops from 4 to 3 and Codex counts as a response — so the PR is approved instead of skipping with waiting-for-advisory-bots. A plain (non-marker) comment is not excluded, so the gate still waits when a real reviewer is genuinely missing.

    Test Results

    bats tests/dev-lead/unit/      → 458 ok / 0 not ok
    bats .../test_advisory_review_gate.bats → 44 ok (5 new, incl. red→green regression for out-of-quota bot)
    bash .dev-lead/scripts/dev-lead-lint.sh → all checks passed
    shellcheck --severity=warning -x scripts/lib/advisory-review-gate.sh → clean
    

    Files Changed

    • scripts/lib/advisory-review-gate.sh: rate-limit marker detection in get_advisory_bot_states; effective_total exclusion logic in check_advisory_reviews
    • tests/dev-lead/unit/test_advisory_review_gate.bats: 5 new tests (marker constant, effective_total, RATE_LIMITED classification, out-of-quota-doesn't-block flip, no-over-exclusion guard)

    Notes

    • Scope limited to fix test issue from agent #1 as the issue instructed ("pick, keep minimal"). Fix Go-live improvements for PR review agent #2 (decouple fallback / anchor head-age) and fix Add @claude delegation, auto-merge, and rebase handling #3 (env escape hatch) were not implemented and remain open follow-ups if needed.
    • A genuinely silent out-of-quota bot (posts nothing) still relies on the existing timeout fallbacks — self-healing only triggers once the bot posts its usage-limit comment, which is the documented behaviour across the cited PRs.
    • The separate infra/billing item (restoring CodeRabbit/Codex prepaid credits) is out of code scope.
  3. added a commit that references this issue on Jun 15, 2026
  4. 6 remaining items

  5. added 12 commits that reference this issue on Aug 2, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reportsdev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions