Repository navigation
Description
1. Executive Summary
Status: DEGRADED
Period: 2026-05-12 10:40 UTC – 2026-05-13 08:07 UTC
Result: 3 of 14 runs failed (21%)
Key findings:
- Root cause of all 3 failures: Copilot fallback engine crashes with
error: Invalid command format— the oldgh copilot suggest -p "$(cat <file>)"invocation is incompatible with moderngh copilotCLI; the session-abort circuit-breaker misclassifies this as a rate limit, skipping 19–22 remaining PRs per run - Amplifier: Gemini fallback is completely unavailable (
GOOGLE_API_KEYunset,@google/gemini-clinot installed), so the only fallback after Claude rate-limits is the broken Copilot engine - Trigger: Claude Opus 4.7 (used for single-pass approval confirmation) hits rate limits during busy windows, forcing the fallback chain that leads to session abort
- Compounding defect: Even in successful runs, the rubber-duck reviewer (
run_duckvia Copilot) silently fails with the sameInvalid command formaterror — it is non-fatal only because the deep-review result is used alone - Pending fix exists but not merged: PR fix: replace gh copilot suggest with GitHub Models REST API #151 (
claude/issue-147-20260512-1802) replaces all threegh copilot suggestcalls with GitHub Models REST API — the fix is in review while the breakage continues in production onmainat SHA22e46ea7f67a57d9bf3de1c7dd656667751c11bf
Action required: Merge PR #151 immediately to restore the Copilot fallback engine and eliminate session aborts.
2. Failure Breakdown
| Failure Category | Affected Runs | Example Error Message |
|---|---|---|
CLI breaking change — gh copilot suggest -p invalid flag |
#720, #721, #722 (100% of failures) | error: Invalid command format. Did you mean: copilot -i "suggest -p # Tier 1: Triage... |
| Engine rate limit — Claude Opus 4.7 quota exhausted | #720, #721, #722 (trigger for fallback) | [approve] rate limit detected — exiting with code 2 for engine fallback |
| Missing fallback — Gemini unavailable | #720, #721, #722 (amplifier) | Claude rate limit hit but Gemini fallback unavailable (CLI not installed or GOOGLE_API_KEY missing) — falling through to Copilot |
| Session abort — no viable fallback engine | #720, #721, #722 (terminal failure) | Rate limit hit on copilot engine, no fallback available for …/pull/151 |
| Silent rubber-duck failure (non-fatal) | #720, #721 (also in successful runs) | [tier2] rubber duck did not produce valid JSON — continuing with deep review only |
3. Error Patterns
Pattern A — Copilot Invalid command format (primary failure)
Exact error (from scripts/engine.sh, run_triage/run_duck copilot branch):
error: Invalid command format.
Did you mean: copilot -i "suggest -p # Tier 1: Triage
...
For non-interactive mode, use the -p or --prompt option.
Try 'copilot --help' for more information.
Step: Review each PR (cascade) → bash scripts/review-batch.sh → scripts/review-one-pr.sh → run_triage()/run_duck() in scripts/engine.sh
Root cause: Three call sites in scripts/engine.sh invoke:
timeout "$TRIAGE_TIMEOUT_SEC" gh copilot suggest -p "$(cat "$prompt_file")"Modern gh copilot no longer accepts -p as a flag for suggest. Additionally, the prompt content begins with # Tier 1: Triage — the # is parsed as a flag prefix, producing Invalid command format. The non-zero exit is then processed by is_rate_limited() in review-one-pr.sh, which matches on the error: prefix and classifies it as a rate limit (exit code 2). This triggers the rate-limit fallback path, and since Gemini is also unavailable, the session aborts.
Pattern B — Gemini fallback unavailable (amplifier)
Exact error (from scripts/review-batch.sh):
##[warning]Claude rate limit hit but Gemini fallback unavailable
(CLI not installed or GOOGLE_API_KEY missing) — falling through to Copilot
##[warning]Gemini rate limit hit — switching to Copilot engine for remaining PRs
Root cause: GOOGLE_API_KEY is empty in every run (GOOGLE_API_KEY: in env dumps). The @google/gemini-cli is installed as best-effort (npm install -g @google/gemini-cli || true) during the Claude engine path, but without an API key, the CLI cannot function. The second warning (Gemini rate limit hit) is emitted immediately after because the code re-enters the fallback chain with REVIEW_ENGINE=gemini and that also exits 2 — directing traffic to Copilot.
Pattern C — Session abort (terminal)
Exact error:
##[error]Rate limit hit on copilot engine, no fallback available for
https://github.com/petry-projects/.github-private/pull/151
##[error]Session aborted early after failure on …/pull/151
(rate-limit on fallback engine). Skipped 21 remaining candidate(s);
will retry on next scheduled run.
Summary: 0 reviews posted, 2 no-ops skipped, 1 failures, 1 engine fallback(s)
to copilot (processed 3/24 candidates) [SESSION ABORTED EARLY]
Root cause: The session-abort guard in scripts/review-batch.sh treats any engine returning exit 2 (rate-limit) when that engine is already the fallback as unrecoverable. Because Copilot exits 2 due to the Invalid command format being misclassified as a rate limit, the guard fires and halts all remaining PR processing.
4. Token Scope Analysis
Scopes currently present (from gh auth status in all run logs):
notifications, read:audit_log, read:discussion, read:org,
read:project, repo, workflow
Assessment: Token scopes are not the cause of any observed failure. The repo scope covers PR reads/writes; read:org covers org membership queries; workflow covers the dispatch mechanism. No permission-denied or 403 errors appear in any log.
| Scope | Status | Notes |
|---|---|---|
repo |
Present | Sufficient for PR reads, comments, approvals |
read:org |
Present | Sufficient for org-member checks |
workflow |
Present | Required for repository_dispatch triggers |
notifications |
Present | Not required but harmless |
read:audit_log |
Present | Not required but harmless |
read:discussion |
Present | Not required but harmless |
read:project |
Present | Not required but harmless |
Recommendation: No token scope changes required. The failures are entirely CLI-compatibility and API-key configuration issues.
5. Recommendations
1. Merge PR #151 immediately
- What: Merge
petry-projects/.github-privatePR fix: replace gh copilot suggest with GitHub Models REST API #151 (fix: replace gh copilot suggest with GitHub Models REST API) - Files changed:
scripts/engine.sh(replaces 3×gh copilot suggest -pwithcopilot_chat()REST API calls),scripts/review-batch.sh(adds Copilot pre-flight smoke test),tests/test_copilot_chat.sh(new unit tests) - Why: Eliminates the
Invalid command formaterror at all three Copilot call sites, restores the Copilot fallback engine, and prevents future session aborts from this cause - Expected impact: All three failure modes (Pattern A, B terminal, C) resolved; Copilot fallback becomes functional
- Urgency: CRITICAL — this is the direct cause of 100% of workflow failures and session aborts skipping 19–22 PRs per affected run
2. Set GOOGLE_API_KEY secret or remove Gemini from the fallback chain
- What: Either add
GOOGLE_API_KEYas a repository secret inpetry-projects/.github-private(Settings → Secrets → Actions), or accept that Gemini is permanently unavailable and document that Copilot is the sole fallback - Files:
pr-review.ymlalready wiresGOOGLE_API_KEY: ${{ secrets.GOOGLE_API_KEY }}; the secret just needs to be populated - Why: Without Gemini, the fallback chain collapses to Claude → (missing) → Copilot. Any Claude rate limit immediately stresses the one remaining fallback. Once PR fix: replace gh copilot suggest with GitHub Models REST API #151 is merged, Copilot will work, but Gemini as a middle tier would prevent Copilot from being the sole safety net
- Expected impact: Three-tier fallback restored; Claude rate-limit events no longer put the entire session at single-point-of-failure risk
- Urgency: HIGH — reduces blast radius of Claude rate limits after the critical fix is applied
3. Fix the rate-limit detector to distinguish CLI errors from actual rate limits
- What: In
scripts/review-one-pr.sh(or whereveris_rate_limited()is defined), tighten the pattern match so thaterror: Invalid command formatis not classified as a rate limit. It should only match on HTTP 429 responses or known rate-limit message strings, not on arbitraryerror:prefixes - Why: The misclassification of CLI errors as rate limits is what converts a recoverable tool error into a session-aborting fallback cascade. Even after PR fix: replace gh copilot suggest with GitHub Models REST API #151 fixes the Copilot invocation, future CLI errors could still trigger false rate-limit exits if the detector remains too broad
- Expected impact: CLI errors fail gracefully (skipping that PR) rather than aborting the entire session
- Urgency: HIGH — defensive hardening; PR fix: replace gh copilot suggest with GitHub Models REST API #151 partially addresses this for Copilot but the underlying detector logic may affect other engines
4. Install @google/gemini-cli unconditionally and validate GOOGLE_API_KEY at startup
- What: Move
npm install -g @google/gemini-cliout of the|| truebest-effort block in the Claude engine case and make it a required install whenGOOGLE_API_KEYis set; add a startup check that emits a clear::warning::if the key is absent. PR fix: add Gemini pre-flight check and engine-availability job summary #150 partially addresses this (engine availability summary) — ensure it is also merged - Files:
.github/workflows/pr-review.yml(Install review engine CLIs step),scripts/validate-engines.sh(added by PR fix: add Gemini pre-flight check and engine-availability job summary #150) - Why: Silent Gemini absence is only discovered mid-run when Claude rate-limits, at which point it is too late to take corrective action
- Expected impact: Operators are warned at run start rather than discovering Gemini is missing during a live rate-limit event
- Urgency: MEDIUM — operational visibility improvement; reduces surprise during incidents
5. Update actions/cache@v4 to a Node.js 24-compatible version before June 2, 2026
- What: In
.github/workflows/pr-review.yml, pinactions/cacheto a release that supports Node.js 24 (e.g.actions/cache@v4.2.0or later that has the Node 24 runtime) - Why: Current
actions/cache@v4(SHA0057852b) will be forced to Node.js 24 on 2026-06-02, and will be removed from runners on 2026-09-16. The warning appears in every run log - Expected impact: Eliminates deprecation warnings; prevents potential cache step breakage post-June 2026
- Urgency: LOW — no current functional impact; deadline is June 2, 2026
6. Health Score
Health: 5/10 — Copilot fallback is broken by a known CLI incompatibility (fix is in PR #151, unmerged), causing complete session aborts that skip 19–22 PRs whenever Claude rate-limits, while the Gemini middle-tier fallback is also absent; successful runs proceed normally but without a functional rubber-duck reviewer.
Activity
don-petry commentedon May 13, 2026
All 5 recommendations resolved:
is_cli_error()rate-limit detector hardening)GOOGLE_API_KEYsecret set and verified (dry-run confirmed no Gemini-unavailable warnings)actions/cachealready on v5