Repository navigation
Description
Summary
The pr-review agent is not failing — it is skipping auto-approval on green PRs with decision: skip, reason: waiting-for-advisory-bots, and the skip never clears. Result: clean, CI-green, rebased PRs sit at REVIEW_REQUIRED indefinitely because donpetry-bot never posts its org-leads approval.
Root cause: the advisory-review-gate waits for advisory bots to submit a review at head, but two of the four advisory bots are out of quota and cannot submit, and the gate does not degrade gracefully when a bot is rate-limited.
Evidence
pr-review run for PR #623 (review/review job 81223083427, run 27478909009):
[advisory-gate] Advisory bots detected: gemini-code-assist sonarqubecloud
[advisory-gate] WARNING: Only 2/4 advisory bots submitted so far (head age: ~60s, last submission: ~Ns ago)
[advisory-gate] WARNING: Will re-check when remaining bots submit their reviews
skip: advisory bots still reviewing (non-blocking, will re-check on bot submission)
{"pr":".../623","decision":"skip","reason":"waiting-for-advisory-bots"}
The two missing/non-responding advisory bots are out of quota — confirmed by their own comments across #479, #483, #516, #518, #520, #525, #550, #580, #621, #622, #623:
- CodeRabbit: "Review limit reached … Your organization has used up its prepaid credits, and credit purchases are no longer available."
- chatgpt-codex-connector: "You have reached your Codex usage limits for code reviews."
Root cause (code)
scripts/lib/advisory-review-gate.sh:
ADVISORY_BOTS(L32–37) =gemini-code-assist,copilot-pull-request-reviewer,sonarqubecloud,chatgpt-codex-connector.check_advisory_reviews(L128–235) requiresnum_submitted == total_advisory_bots(all 4) or one of two timeout fallbacks: head-commit age > 1200s (L222) or quiescence > 600s (L224). Otherwisereturn 1→ pr-review skips (review-one-pr.shL140–177,reason=waiting-for-advisory-bots).- In practice the gate rarely sees 4/4: Codex is permanently out of quota, and Copilot reviews only a subset of PRs. So approval depends entirely on the timeout fallbacks — which only re-evaluate on an event re-trigger (a bot submission) or the hourly stuck-review sweep. Every dev-lead push (e.g. a rebase) resets the head-age clock to 0, so freshly-rebased PRs restart the 20-min wait and keep skipping.
- A
RATE_LIMITEDdisplay state already exists (format_bot_statusL110) but nothing ever sets it —get_advisory_bot_states(L59–85) only maps a bot's literal review/comment state. So the half-built rate-limit handling never triggers.
Impact
This is the gate behind the current backlog of green-CI PRs stuck at REVIEW_REQUIRED (e.g. #479, #483, #516, #518, #520, #525, #550, #589, #620, #621, #622, #623). With CodeRabbit + Codex credits exhausted, the advisory reviews won't arrive, so the gate only clears via the 20-min/10-min fallbacks after a quiet period + a re-trigger — slow and fragile, and reset by any new push.
Recommended fix (pick, keep minimal + tests)
- Wire up
RATE_LIMITED(preferred). Inget_advisory_bot_states, detect a bot comment whose body matches the known rate-limit/usage-limit markers (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits") and emitstate: RATE_LIMITED. TreatRATE_LIMITED(andUNSUPPORTED) bots as responded/non-participating so they don't hold the gate. This makes the gate degrade gracefully the moment a bot signals it's out. - Shorten / decouple the fallback so a bounded wait reliably proceeds even without a re-trigger (pair with the sweep), and/or anchor head-age so a no-op rebase doesn't reset the clock.
- Config escape hatch: allow dropping a bot from the required advisory set (keep it advisory-only) via env, so an out-of-quota bot (Codex) can be removed without code change.
Separate (infra/billing — not code)
CodeRabbit and Codex have exhausted prepaid credits org-wide, so they aren't reviewing at all right now. Even with a perfect gate, restoring those credits (or accepting their absence via fix #1/#3) is needed for advisory coverage.
Filed from a triage of why pr-review wasn't approving green PRs. Evidence: run 27478909009 (#623) + rate-limit comments on the PRs listed above.
Activity
don-petry commentedon Jun 13, 2026
Dev-Lead Implementation Plan
Issue: #657 — pr-review never auto-approves when advisory bots are rate-limited/out-of-quota
Scope
Implement recommended fix #1 (preferred): wire up the half-built
RATE_LIMITEDstate inscripts/lib/advisory-review-gate.shso an out-of-quota advisory bot that posts a usage-limit comment is detected, classifiedRATE_LIMITED, and treated as non-participating — dropping it from the required advisory set so it can no longer hold the gate open indefinitely.Implementation Checklist
scripts/lib/advisory-review-gate.sh— add known rate-limit/usage-limit marker regex (CodeRabbit "Review limit reached"/"used up its prepaid credits", Codex "reached your Codex usage limits")scripts/lib/advisory-review-gate.sh—get_advisory_bot_states: emitstate: RATE_LIMITEDfor an advisory-bot comment whose body matches a markerscripts/lib/advisory-review-gate.sh—check_advisory_reviews: countRATE_LIMITED/UNSUPPORTEDbots as responded and subtract them from the required total (effective_total, clamped to ≥1) so a permanently out-of-quota bot doesn't block approvalTests to Write (TDD, in tests/dev-lead/unit/test_advisory_review_gate.bats)
RATE_LIMITEDin gate outputRATE_LIMITEDdetection +effective_totalTest Command
bats tests/dev-lead/unit/(full repo lint:bash .dev-lead/scripts/dev-lead-lint.sh)don-petry commentedon Jun 13, 2026
Dev-Lead: Implementation Complete
Implemented recommended fix #1 (preferred) — wired up the half-built
RATE_LIMITEDstate so an out-of-quota advisory bot can no longer hold the gate open.Plan Execution
RATE_LIMIT_MARKERSregex (CodeRabbit "Review limit reached"/"used up its prepaid credits"; Codex "reached your Codex usage limits")get_advisory_bot_states: a comment from an advisory bot whose body matches a marker is now classifiedstate: RATE_LIMITED(case-insensitive) instead ofCOMMENTEDcheck_advisory_reviews:RATE_LIMITED/UNSUPPORTEDbots are treated as responded and subtracted from the required total (effective_total = total − unavailable, clamped to ≥1); the approval threshold now compares againsteffective_totalBehaviour
With Codex out of quota (posts its usage-limit comment) and Copilot absent, two genuine advisory reviews (Gemini + SonarCloud) now satisfy the gate immediately —
effective_totaldrops from 4 to 3 and Codex counts as a response — so the PR is approved instead of skipping withwaiting-for-advisory-bots. A plain (non-marker) comment is not excluded, so the gate still waits when a real reviewer is genuinely missing.Test Results
Files Changed
scripts/lib/advisory-review-gate.sh: rate-limit marker detection inget_advisory_bot_states;effective_totalexclusion logic incheck_advisory_reviewstests/dev-lead/unit/test_advisory_review_gate.bats: 5 new tests (marker constant,effective_total, RATE_LIMITED classification, out-of-quota-doesn't-block flip, no-over-exclusion guard)Notes
feat: implement issue #657 — pr-review never auto-approves when advis…
feat: implement issue #1182 — F1: record major-scoped-channels decisi…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
chore: major-scope repin add-to-project.yml [#657 F5]
chore: major-scope repin agent-shield.yml [#657 F5]
chore: major-scope repin auto-rebase.yml [#657 F5]
chore: major-scope repin dependabot-automerge.yml [#657 F5]
chore: major-scope repin dependency-audit.yml [#657 F5]
chore: major-scope repin dev-lead.yml [#657 F5]
chore: major-scope repin feature-ideation.yml [#657 F5]
6 remaining items
chore(ci-failure-analyst): re-pin lock stub to v-form channel (@ci-fa…
chore(ci-failure-analyst): re-pin lock stub to v-form channel (@ci-fa…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
chore(ci-failure-analyst): re-pin lock stub to v-form channel (@ci-fa…
feat: implement issue #1184 — F2: cut-release.sh learns v<major>-<tie…
chore(dev-lead): repin stub from dev-lead/v1-stable to dev-lead/v139-…