Repository navigation
Description
1. Executive Summary
Status: DEGRADED
Period: 2026-06-01T12:23Z – 2026-06-02T03:14Z
Result: 2 of 19 runs failed (10.5%); 8 cancelled (concurrency queueing or early-exit conditions)
Key findings:
- All 3 review engines exhausted in sequence (Claude rate-limited → Gemini credits depleted → Copilot CLI hard exit 1) causing both failures
- Gemini
GOOGLE_API_KEYprepayment credits fully depleted — fallback engine is permanently broken until topped up - Copilot triage exits with code 1 with no usable fallback;
COPILOT_API_MODELunbound variable inengine.sh:242signals a missing env config
Action required: Replenish Google AI Studio prepaid credits and diagnose gh copilot triage exit-1 (last-resort fallback is broken).
2. Failure Breakdown
| Failure Category | Affected Runs | Example Error Message |
|---|---|---|
| Engine rate limit — Claude quota | #2495, #2497 | You've hit your limit · resets 5:10pm (UTC) |
| Engine rate limit — Gemini credits depleted | #2495, #2497 | Your prepayment credits are depleted … RESOURCE_EXHAUSTED |
| Copilot CLI hard failure (last fallback) | #2495, #2497 | triage exited with code 1 / cascade failed at tier 1 (triage process exit 1) |
| Unbound variable in engine.sh | #2495 | engine.sh: line 242: COPILOT_API_MODEL: unbound variable |
3. Error Patterns
Pattern 1 — Claude rate limit (both runs)
[tier1] usage/rate limit detected on stdout — exiting with code 2 for engine fallback
limit message: You've hit your limit · resets 5:10pm (UTC)
- Step:
Review each PR (cascade)→scripts/review-batch.sh→scripts/review-one-pr.shtier-1 triage - Root cause: Claude API OAuth token (
CLAUDE_CODE_OAUTH_TOKEN) hit its per-day usage ceiling. This is a time-bounded limit (resets at ~17:10 UTC); both failures occurred between 12:37–13:47 UTC. The rate limit was triggered on the very firstclaude-haiku-4-5triage call in both runs, suggesting the daily budget was already exhausted from earlier runs in the day.
Pattern 2 — Gemini prepayment credits exhausted (both runs)
"code": 429,
"message": "Your prepayment credits are depleted. Please go to AI Studio …"
"status": "RESOURCE_EXHAUSTED"
Attempt 10 failed: … Max attempts reached
- Step: Gemini fallback engine triage (
gemini-2.0-flash) - Root cause: The
GOOGLE_API_KEY/GEMINI_API_KEYis billed against a prepaid AI Studio credit balance, and that balance reached zero. The Gemini CLI retried 10 times with exponential backoff, all returning HTTP 429, then terminated. This is a billing exhaustion — not a rate-limit reset — so the fallback remains broken until credits are replenished.
Pattern 3 — Copilot triage hard exit 1 (both runs)
[tier1] triage (o4-mini)
##[warning]triage exited with code 1
triage stderr: [copilot] calling gh copilot (model=openai/o4-mini, timeout=300s, flags=--deny-tool *)
##[error]cascade failed at tier 1 (triage process exit 1)
- Step: Copilot last-resort fallback, tier-1 triage
- Root cause:
gh copilotexits non-zero immediately after being called. The only stderr captured is the invocation line itself — no API error is returned — indicatingCOPILOT_GITHUB_TOKENeither lacks a Copilot subscription, has insufficient API scopes, orgh copilotis not properly authenticated with that token. Because the copilot engine is the terminal fallback andcancel-in-progress: falsemeans no retry, this exit-1 terminates the entire run.
Pattern 4 — Unbound variable COPILOT_API_MODEL (run #2495)
/home/runner/work/.github-private/.github-private/scripts/engine.sh: line 242: COPILOT_API_MODEL: unbound variable
- Step:
Review each PR (cascade)→scripts/engine.shline 242, during the deep-review + duck phase (tier 2) before the fallback chain - Root cause:
engine.shreferences$COPILOT_API_MODELwithset -uactive but the variable is never exported in the workflow env or set in the script. This is a script bug — the variable is referenced but has no default value guard (e.g.,${COPILOT_API_MODEL:-}or a prior assignment). In run pr-review: Fable adjudication tier before escalating to a human #2495 this occurred mid-tier-2 but did not immediately abort the cascade; the run eventually failed at the copilot triage stage.
4. Token Scope Analysis
Scopes currently present (from gh auth status in logs):
| Token | Identity | Observed Permissions |
|---|---|---|
GH_TOKEN (DON_PETRY_BOT_GH_PAT) |
donpetry-bot |
checks:read, contents:read, metadata:read, pull-requests:write |
COPILOT_GITHUB_TOKEN (GH_PAT) |
Unknown (separate user PAT) | Not shown in logs |
CLAUDE_CODE_OAUTH_TOKEN |
Anthropic OAuth | Claude usage quota (per-day, hit at ~12:37 UTC) |
GOOGLE_API_KEY / GEMINI_API_KEY |
Google AI Studio prepaid | Credits depleted |
Scopes appearing missing or insufficient:
| Token | Missing/Insufficient | Evidence |
|---|---|---|
COPILOT_GITHUB_TOKEN |
Copilot access not confirmed — token may lack copilot scope or the associated account has no active Copilot subscription |
gh copilot exits 1 immediately on auth; no API error returned (auth failures behave this way with gh copilot) |
GOOGLE_API_KEY |
Billing not active — prepayment balance at zero | HTTP 429 RESOURCE_EXHAUSTED with "prepayment credits are depleted" |
Recommendations per token:
COPILOT_GITHUB_TOKEN: Verify the PAT's account has an active GitHub Copilot for Business or Enterprise subscription. The token requirescopilotscope (classic PAT) or the fine-grained equivalent. Runenv GH_TOKEN=<token> gh copilot --versionlocally to confirm auth works before the next run.GOOGLE_API_KEY: Top up the prepaid credit balance at AI Studio. Consider switching to a monthly billing account to avoid hard credit exhaustion; prepay balances create a cliff failure rather than a gradual rate-limit.
5. Recommendations
1. Replenish Google AI Studio prepaid credits [CRITICAL]
- What: Navigate to AI Studio billing for the project associated with
GOOGLE_API_KEYand add prepaid credits (or migrate to monthly billing). - Why: Both failed runs exhausted the Gemini fallback entirely. With Claude also rate-limited, the only remaining fallback is Copilot. If Copilot fails too, the run dies. Gemini credits depletion is a hard, persistent failure unlike a rate-limit reset.
- Expected impact: Restores the Gemini fallback engine, reducing blast radius of Claude rate-limit events from 100% failure to graceful fallback.
- Urgency: CRITICAL
2. Fix COPILOT_API_MODEL unbound variable in scripts/engine.sh:242 [HIGH]
- What: In
scripts/engine.shat line 242, change the bare reference$COPILOT_API_MODELto${COPILOT_API_MODEL:-openai/o4-mini}(or set a default earlier in the file). Alternatively, addCOPILOT_API_MODEL="${COPILOT_API_MODEL:-openai/o4-mini}"before line 242. - Why: With
set -uactive, any code path that reaches line 242 without the variable set causes an unbound variable abort. This silently swallows the tier-2 duck result and may mask deeper failures. - Expected impact: Eliminates spurious script aborts; duck review output is preserved for the cascade.
- Urgency: HIGH
3. Diagnose and fix gh copilot triage exit-1 [HIGH]
- What: Run
env GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot suggest "hello" 2>&1in a test workflow step and capture the full output. If auth fails, add thecopilotscope to the PAT or ensure the token's account has a Copilot subscription. Add a diagnostic step before the cascade that callsenv GH_TOKEN="$COPILOT_GITHUB_TOKEN" gh copilot --versionand emits a::warning::if it fails, rather than silently breaking the last fallback. - Why:
gh copilotis the only remaining engine when Claude + Gemini are both unavailable. Currently it exits 1 with only the invocation line in stderr — there is no diagnostic information captured. This makes debugging opaque and leaves the system with zero fallbacks. - Expected impact: Either restores Copilot as a working last-resort fallback, or surfaces a clear error before entering the copilot engine path so the run can fail fast with an actionable message.
- Urgency: HIGH
4. Add a Claude rate-limit guard / schedule review runs after reset time [MEDIUM]
- What: In the workflow env, add an optional
CLAUDE_RESET_HOUR_UTCvariable (e.g.,17). Inreview-batch.sh, if Claude returns exit code 2, log the reset time and consider whether to proceed with fallbacks or park the run until reset. Alternatively, the rate-limit reset time (resets 5:10pm UTC) is consistent — consider scheduling high-volume review windows after 17:10 UTC. - Why: Both failures started at ~12:37 UTC, well before the 17:10 UTC reset. Claude was exhausted earlier in the day. If fallbacks were healthy this would be fine, but chaining through broken fallbacks wastes ~8 minutes per run and still fails.
- Expected impact: Reduces wasted runner time and fallback engine load on days when Claude's daily quota is exhausted.
- Urgency: MEDIUM
5. Add Gemini billing alert / health-check before using the engine [MEDIUM]
- What: In
scripts/review-batch.shor the engine dispatcher, before invoking Gemini, run a lightweight probe (e.g.,gemini --versionor a minimal API call) and check forRESOURCE_EXHAUSTED. If detected, skip Gemini immediately and emit::warning::rather than running 10 retry attempts (~3.5 minutes wasted per run). - Why: Gemini prepayment exhaustion is persistent (not time-resetting), so retrying 10 times with backoff wastes significant runner time on a condition that cannot self-heal.
- Expected impact: Eliminates ~3-4 minutes of wasted retry time per run when Gemini credits are depleted; surfaces billing issue immediately.
- Urgency: MEDIUM
6. Health Score
Health: 5/10 — System recovers when Claude's daily rate limit resets, but both fallback engines (Gemini credits depleted, Copilot CLI broken) are non-functional, leaving zero resilience when Claude is unavailable.
Activity
don-petry commentedon Jul 14, 2026
Closing stale daily digest — superseded by the newest snapshot. Digest steps now close prior days automatically (#1204 / #1212).