Skip to content

PR Review Agent — failures detected 2026-06-02 #418

Closed

Description

@github-actions

1. Executive Summary

Status: DEGRADED
Period: 2026-06-01T12:23Z – 2026-06-02T03:14Z
Result: 2 of 19 runs failed (10.5%); 8 cancelled (concurrency queueing or early-exit conditions)

Key findings:

  • All 3 review engines exhausted in sequence (Claude rate-limited → Gemini credits depleted → Copilot CLI hard exit 1) causing both failures
  • Gemini GOOGLE_API_KEY prepayment credits fully depleted — fallback engine is permanently broken until topped up
  • Copilot triage exits with code 1 with no usable fallback; COPILOT_API_MODEL unbound variable in engine.sh:242 signals a missing env config

Action required: Replenish Google AI Studio prepaid credits and diagnose gh copilot triage exit-1 (last-resort fallback is broken).


2. Failure Breakdown

Failure Category Affected Runs Example Error Message
Engine rate limit — Claude quota #2495, #2497 You've hit your limit · resets 5:10pm (UTC)
Engine rate limit — Gemini credits depleted #2495, #2497 Your prepayment credits are depleted … RESOURCE_EXHAUSTED
Copilot CLI hard failure (last fallback) #2495, #2497 triage exited with code 1 / cascade failed at tier 1 (triage process exit 1)
Unbound variable in engine.sh #2495 engine.sh: line 242: COPILOT_API_MODEL: unbound variable

3. Error Patterns

Pattern 1 — Claude rate limit (both runs)

[tier1] usage/rate limit detected on stdout — exiting with code 2 for engine fallback
limit message: You've hit your limit · resets 5:10pm (UTC)
  • Step: Review each PR (cascade) → scripts/review-batch.sh → scripts/review-one-pr.sh tier-1 triage
  • Root cause: Claude API OAuth token (CLAUDE_CODE_OAUTH_TOKEN) hit its per-day usage ceiling. This is a time-bounded limit (resets at ~17:10 UTC); both failures occurred between 12:37–13:47 UTC. The rate limit was triggered on the very first claude-haiku-4-5 triage call in both runs, suggesting the daily budget was already exhausted from earlier runs in the day.

Pattern 2 — Gemini prepayment credits exhausted (both runs)

"code": 429,
"message": "Your prepayment credits are depleted. Please go to AI Studio …"
"status": "RESOURCE_EXHAUSTED"
Attempt 10 failed: … Max attempts reached
  • Step: Gemini fallback engine triage (gemini-2.0-flash)
  • Root cause: The GOOGLE_API_KEY / GEMINI_API_KEY is billed against a prepaid AI Studio credit balance, and that balance reached zero. The Gemini CLI retried 10 times with exponential backoff, all returning HTTP 429, then terminated. This is a billing exhaustion — not a rate-limit reset — so the fallback remains broken until credits are replenished.

Pattern 3 — Copilot triage hard exit 1 (both runs)

[tier1] triage (o4-mini)
##[warning]triage exited with code 1
    triage stderr:     [copilot] calling gh copilot (model=openai/o4-mini, timeout=300s, flags=--deny-tool *)
##[error]cascade failed at tier 1 (triage process exit 1)
  • Step: Copilot last-resort fallback, tier-1 triage
  • Root cause: gh copilot exits non-zero immediately after being called. The only stderr captured is the invocation line itself — no API error is returned — indicating COPILOT_GITHUB_TOKEN either lacks a Copilot subscription, has insufficient API scopes, or gh copilot is not properly authenticated with that token. Because the copilot engine is the terminal fallback and cancel-in-progress: false means no retry, this exit-1 terminates the entire run.

Pattern 4 — Unbound variable COPILOT_API_MODEL (run #2495)

/home/runner/work/.github-private/.github-private/scripts/engine.sh: line 242: COPILOT_API_MODEL: unbound variable
  • Step: Review each PR (cascade) → scripts/engine.sh line 242, during the deep-review + duck phase (tier 2) before the fallback chain
  • Root cause: engine.sh references $COPILOT_API_MODEL with set -u active but the variable is never exported in the workflow env or set in the script. This is a script bug — the variable is referenced but has no default value guard (e.g., ${COPILOT_API_MODEL:-} or a prior assignment). In run pr-review: Fable adjudication tier before escalating to a human #2495 this occurred mid-tier-2 but did not immediately abort the cascade; the run eventually failed at the copilot triage stage.

4. Token Scope Analysis

Scopes currently present (from gh auth status in logs):

Token Identity Observed Permissions
GH_TOKEN (DON_PETRY_BOT_GH_PAT) donpetry-bot checks:read, contents:read, metadata:read, pull-requests:write
COPILOT_GITHUB_TOKEN (GH_PAT) Unknown (separate user PAT) Not shown in logs
CLAUDE_CODE_OAUTH_TOKEN Anthropic OAuth Claude usage quota (per-day, hit at ~12:37 UTC)
GOOGLE_API_KEY / GEMINI_API_KEY Google AI Studio prepaid Credits depleted

Scopes appearing missing or insufficient:

Token Missing/Insufficient Evidence
COPILOT_GITHUB_TOKEN Copilot access not confirmed — token may lack copilot scope or the associated account has no active Copilot subscription gh copilot exits 1 immediately on auth; no API error returned (auth failures behave this way with gh copilot)
GOOGLE_API_KEY Billing not active — prepayment balance at zero HTTP 429 RESOURCE_EXHAUSTED with "prepayment credits are depleted"

Recommendations per token:

  • COPILOT_GITHUB_TOKEN: Verify the PAT's account has an active GitHub Copilot for Business or Enterprise subscription. The token requires copilot scope (classic PAT) or the fine-grained equivalent. Run env GH_TOKEN=<token> gh copilot --version locally to confirm auth works before the next run.
  • GOOGLE_API_KEY: Top up the prepaid credit balance at AI Studio. Consider switching to a monthly billing account to avoid hard credit exhaustion; prepay balances create a cliff failure rather than a gradual rate-limit.

5. Recommendations

1. Replenish Google AI Studio prepaid credits [CRITICAL]

  • What: Navigate to AI Studio billing for the project associated with GOOGLE_API_KEY and add prepaid credits (or migrate to monthly billing).
  • Why: Both failed runs exhausted the Gemini fallback entirely. With Claude also rate-limited, the only remaining fallback is Copilot. If Copilot fails too, the run dies. Gemini credits depletion is a hard, persistent failure unlike a rate-limit reset.
  • Expected impact: Restores the Gemini fallback engine, reducing blast radius of Claude rate-limit events from 100% failure to graceful fallback.
  • Urgency: CRITICAL

2. Fix COPILOT_API_MODEL unbound variable in scripts/engine.sh:242 [HIGH]

  • What: In scripts/engine.sh at line 242, change the bare reference $COPILOT_API_MODEL to ${COPILOT_API_MODEL:-openai/o4-mini} (or set a default earlier in the file). Alternatively, add COPILOT_API_MODEL="${COPILOT_API_MODEL:-openai/o4-mini}" before line 242.
  • Why: With set -u active, any code path that reaches line 242 without the variable set causes an unbound variable abort. This silently swallows the tier-2 duck result and may mask deeper failures.
  • Expected impact: Eliminates spurious script aborts; duck review output is preserved for the cascade.
  • Urgency: HIGH

3. Diagnose and fix gh copilot triage exit-1 [HIGH]

  • What: Run env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot suggest "hello" 2>&1 in a test workflow step and capture the full output. If auth fails, add the copilot scope to the PAT or ensure the token's account has a Copilot subscription. Add a diagnostic step before the cascade that calls env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --version and emits a ::warning:: if it fails, rather than silently breaking the last fallback.
  • Why: gh copilot is the only remaining engine when Claude + Gemini are both unavailable. Currently it exits 1 with only the invocation line in stderr — there is no diagnostic information captured. This makes debugging opaque and leaves the system with zero fallbacks.
  • Expected impact: Either restores Copilot as a working last-resort fallback, or surfaces a clear error before entering the copilot engine path so the run can fail fast with an actionable message.
  • Urgency: HIGH

4. Add a Claude rate-limit guard / schedule review runs after reset time [MEDIUM]

  • What: In the workflow env, add an optional CLAUDE_RESET_HOUR_UTC variable (e.g., 17). In review-batch.sh, if Claude returns exit code 2, log the reset time and consider whether to proceed with fallbacks or park the run until reset. Alternatively, the rate-limit reset time (resets 5:10pm UTC) is consistent — consider scheduling high-volume review windows after 17:10 UTC.
  • Why: Both failures started at ~12:37 UTC, well before the 17:10 UTC reset. Claude was exhausted earlier in the day. If fallbacks were healthy this would be fine, but chaining through broken fallbacks wastes ~8 minutes per run and still fails.
  • Expected impact: Reduces wasted runner time and fallback engine load on days when Claude's daily quota is exhausted.
  • Urgency: MEDIUM

5. Add Gemini billing alert / health-check before using the engine [MEDIUM]

  • What: In scripts/review-batch.sh or the engine dispatcher, before invoking Gemini, run a lightweight probe (e.g., gemini --version or a minimal API call) and check for RESOURCE_EXHAUSTED. If detected, skip Gemini immediately and emit ::warning:: rather than running 10 retry attempts (~3.5 minutes wasted per run).
  • Why: Gemini prepayment exhaustion is persistent (not time-resetting), so retrying 10 times with backoff wastes significant runner time on a condition that cannot self-heal.
  • Expected impact: Eliminates ~3-4 minutes of wasted retry time per run when Gemini credits are depleted; surfaces billing issue immediately.
  • Urgency: MEDIUM

6. Health Score

Health: 5/10 — System recovers when Claude's daily rate limit resets, but both fallback engines (Gemini credits depleted, Copilot CLI broken) are non-functional, leaving zero resilience when Claude is unavailable.

Activity

  1. don-petry commented on Jul 14, 2026

    @don-petry
    Collaborator

    Closing stale daily digest — superseded by the newest snapshot. Digest steps now close prior days automatically (#1204 / #1212).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions