Repository navigation
Description
PR Review Agent — Health Report
Generated: 2026-05-11 | Window: 2026-05-10T09:00Z – 2026-05-11T06:47Z
1. Executive Summary
Status: DEGRADED
Period: 2026-05-10T09:00Z – 2026-05-11T06:47Z
Result: 16 of 29 runs failed (55%)
Key findings:
- 10 early failures (runs [Phase 3] Cut ci-failure-analyst v1.0.0 + stable tags and extend the channel ruleset #672–[Phase 2] Pilot MCP-enriched review on one downstream repo and measure quality + latency #681) caused by wrong
GH_TOKENidentity —don-petry's own PAT then a fine-grained PAT replaced it; both are now resolved (classic PAT fordonpetry-botin use since run initiative-planner: Bob materializes a plan even when he has unresolved open questions — should gate, not guess #682) broodly/pull/30tier-3 security audit exits code 1 on every encounter, aborting the entire batch session and skipping remaining candidates (confirmed in runs initiative-planner: Bob materializes a plan even when he has unresolved open questions — should gate, not guess #682, [Fleet Monitor] petry-projects/.github-private — .github/workflows/dependency-audit.yml #700)- Rubber duck (Copilot CLI cross-engine check) is fully broken on all recent runs — wrong
gh copilotflag syntax; non-fatal but cross-engine review is silently disabled - Intermittent AI-tier instability: 4 failures from Claude usage cap, invalid JSON responses, and an unrecognized
"skip"decision value from the confirmation tier
Action required: Identify and resolve the tier-3 audit failure on broodly/pull/30 — it is the only active session-abort trigger in current runs.
2. Failure Breakdown
| Failure Category | Affected Runs | Example Error Message |
|---|---|---|
| Wrong bot identity — self-approval | #672, #673, #674, #675, #676, #677, #678 (7 runs) | GraphQL: Review Can not approve your own pull request (addPullRequestReview) |
| Fine-grained PAT — missing PR write | #679, #680, #681 (3 runs) | GraphQL: Resource not accessible by personal access token (addPullRequestReview) |
| Per-PR tier-3 audit hard failure | #682, #700 (2 runs) | ##[error]Review failed for …broodly/pull/30 (exit code 1) |
| Claude usage cap → invalid AI response | #689, #691 (2 runs) | You're out of extra usage · resets 3:30am (UTC) / single-review did not produce valid JSON |
| Engine rate limit + Copilot CLI broken | #693 (1 run) | error: Invalid command format. Did you mean: copilot -i "suggest …" |
| Invalid decision value from AI tier | #695 (1 run) | ERROR: invalid decision 'skip' |
| Rubber duck non-fatal (CLI syntax) | All runs #678 onward (ongoing) | error: unknown option '--target' / Invalid command format |
3. Error Patterns
Pattern A — Wrong bot identity / self-approval (runs #672–#678)
failed to create review: GraphQL: Review Can not approve your own pull request (addPullRequestReview)
ERROR: gh pr review failed with exit code 1
Step: bash scripts/review-batch.sh → final gh pr review call
Root cause: GH_TOKEN was don-petry's personal classic PAT, not a bot PAT. The first candidate PR was don-petry/self-private/pull/1, authored by don-petry. GitHub forbids approving one's own PR. The token identity also lacked read:org. This affected every hourly run until the token was rotated.
Pattern B — Fine-grained PAT restriction (runs #679–#681)
failed to create review: GraphQL: Resource not accessible by personal access token (addPullRequestReview)
ERROR: gh pr review failed with exit code 1
Step: bash scripts/review-batch.sh → gh pr review call
Root cause: A fine-grained PAT (github_pat_11CDFSYKQ0bBnkc5WYzct1_…) was used for DON_PETRY_BOT_GH_PAT. Fine-grained PATs require pull-requests: write to be explicitly granted per repository; the GraphQL addPullRequestReview mutation was blocked. Resolved in run #682 when a classic PAT (ghp_…) was deployed.
Pattern C — broodly/pull/30 tier-3 security audit hard-fail (runs #682, #700)
[tier3] security audit (claude-opus-4-6)
##[error]Review failed for https://github.com/petry-projects/broodly/pull/30 (exit code 1)
##[error]Session aborted early after failure on …broodly/pull/30 (exit code 1).
Skipped 11–12 remaining candidate(s); will retry on next scheduled run.
Step: bash scripts/review-one-pr.sh → tier-3 (claude-opus-4-6) audit block
Root cause: The security audit for this PR exits with code 1 (hard failure, not code 2 for rate-limit). The audit runs for ~75 seconds before failing. The PR has 29,210 additions across 127 files with HIGH-risk signals (IDOR vulnerabilities, SonarCloud failure, merge conflicts). Possible causes: Opus rate-limit not caught by is_rate_limited in the tier-3 code path (PR #107 only patched tier-1), malformed output from the audit agent, or the per-PR script exits 1 on a non-JSON verdict. Every time this PR surfaces it aborts the entire batch session.
Pattern D — Claude usage cap → invalid deep/confirmation JSON (runs #689, #691)
deep review did not produce valid JSON at /tmp/cascade/deep.json
You're out of extra usage · resets 3:30am (UTC) [run #689]
single-review did not produce valid JSON [run #691]
##[error]cascade failed at tier 2 / Review failed (exit code 1)
Step: scripts/review-one-pr.sh → tier-2 deep reviewer or single-confirmation claude-opus-4-6
Root cause: Both failures occurred between 02:58Z–03:32Z when the daily Claude usage cap was active ("resets 3:30am"). The code at those SHAs predates PR #107; is_rate_limited did not match billing-cap phrasing. Post-merge (#700, SHA 28a5cb1), tier-1 now detects and falls back, but tier-2/tier-3 rate-limit detection may still lack the same coverage.
Pattern E — Copilot CLI wrong flag syntax (all runs #678 onward, non-fatal)
error: unknown option '--target'
(Did you mean --agent?) [runs #672–#678 era]
error: Invalid command format.
Did you mean: copilot -i "suggest -p # Cross-engine adversarial review…"?
For non-interactive mode, use the -p or --prompt option. [runs #682+]
Step: scripts/review-one-pr.sh → rubber duck (tier-2 copilot cross-check)
Root cause: The gh copilot extension changed its CLI interface. The script calls gh copilot suggest --target shell "…" (old) or gh copilot suggest -p "…" (intermediate), but the current CLI requires gh copilot -i "…". The workflow degrades gracefully (rubber duck did not produce valid JSON — continuing with deep review only), so this is non-fatal, but the adversarial cross-engine check is effectively disabled on every run.
Pattern F — Invalid decision 'skip' from confirmation tier (run #695)
ERROR: invalid decision 'skip'
##[error]Review failed for …/.github-private/pull/107 (exit code 1)
Step: scripts/review-one-pr.sh → single confirmation (claude-opus-4-6) after triage cleared
Root cause: PR #107 was the provider-limit fallback fix PR itself. When Opus confirmed it, it returned "decision": "skip" — a value not in the accepted set (approve/escalate/request-changes). The parser treats unrecognized decisions as fatal. This may be model behaviour when reviewing workflow-automation code that references itself, or a prompt edge case.
4. Token Scope Analysis
Run #672–#678 — don-petry personal PAT
| Scope | Status |
|---|---|
repo |
Present |
workflow |
Present |
read:org |
Missing — gh auth status explicitly warns |
| Bot identity | Wrong — token belongs to the PR author, not a bot account |
Recommendation: Rotate DON_PETRY_BOT_GH_PAT to a token owned by donpetry-bot with read:org. (Already resolved in run #682.)
Run #679–#681 — donpetry-bot fine-grained PAT (github_pat_11CDFSYKQ0bBnkc5WYzct1_)
| Scope | Status |
|---|---|
pull-requests: write |
Absent — GraphQL addPullRequestReview blocked |
| Repository access | Unclear — may be limited to specific repos |
Recommendation: Replace fine-grained PAT with a classic PAT (ghp_) for donpetry-bot that includes repo and read:org. Fine-grained PATs require per-repository explicit grants and may not work across all repos in petry-projects/* and don-petry/*. (Already resolved in run #682.)
Run #682 onward — donpetry-bot classic PAT (ghp_…)
| Scope | Status |
|---|---|
notifications |
Present |
read:audit_log |
Present |
read:discussion |
Present |
read:org |
Present |
read:project |
Present |
repo |
Present |
workflow |
Present |
Assessment: Sufficient for current operations. No missing scopes identified. The addPullRequestReview mutation succeeds.
5. Recommendations
1. Resolve the broodly/pull/30 tier-3 audit failure [CRITICAL]
- What: Investigate why
scripts/review-one-pr.shexits code 1 (not 2) from the opus security audit. Check ifis_rate_limitedis called for tier-3 stderr/stdout in the same way PR fix(pr-review): detect provider usage-cap and fall back to next engine #107 patched tier-1. If it is a rate-limit, the code path likely needs the same dual-channel check added to the opus audit block. - Why: Every time this PR appears, it aborts the entire batch session and skips 11–12 remaining candidates. It has appeared in 2 of the last 10 batch runs.
- Expected impact: Eliminates the main current session-abort trigger; remaining candidates will be processed.
- Urgency: CRITICAL — this is the sole active session-aborting failure after bot identity was fixed.
2. Extend rate-limit detection to tier-2 and tier-3 Claude invocations [HIGH]
- What: In
scripts/review-one-pr.sh, apply the same dual-channelis_rate_limitedpattern (stdout + stderr) that PR fix(pr-review): detect provider usage-cap and fall back to next engine #107 added to the triage path to the deep-review and opus-audit invocations. Currently those tiers exit with code 1 on a usage cap, which aborts the session rather than falling back. - Why: Runs [Phase 2] Repin ci-failure-analyst callers/template/docs to @ci-failure-analyst/stable #689 and [Guardrail] Held-out hygiene: dev/test split so the proposer never scores against cases it can see #691 show deep review and confirmation tiers returning invalid JSON on usage-cap hits; those would have recovered gracefully with exit code 2.
- Expected impact: Rate-limit hits in tier-2/3 trigger engine fallback instead of session abort.
- Urgency: HIGH — daily usage-cap window causes hard failures at peak hours.
3. Fix the rubber duck Copilot CLI command syntax [HIGH]
- What: In the script that invokes the rubber duck (tier-2 copilot cross-check, likely
scripts/run-duck.shor inline inreview-one-pr.sh), update thegh copilotinvocation fromgh copilot suggest -p "…"to the current syntax:gh copilot -i "…"(non-interactive /--promptform). Validate withgh copilot --help. - Why: The cross-engine adversarial review has never produced valid output in any run within this analysis window. The feature is completely disabled in practice despite appearing in logs as attempted.
- Expected impact: Restores the rubber duck second opinion on every tier-2 escalation.
- Urgency: HIGH — the feature is silently non-functional on every run.
4. Harden the confirmation tier decision parser [MEDIUM]
- What: In
scripts/review-one-pr.sh, at the point wheresingle-reviewJSON is parsed, add"skip"as a handled fallback (treat as"escalate") rather than hard-failing with exit code 1. Also log the full model response when the decision is unrecognised to aid debugging. - Why: Run [Fleet Monitor] petry-projects/.github-private — .github/workflows/lint.yml #695 showed Opus returning
"skip"when reviewing an automation PR; this is an edge case that should degrade gracefully (escalate to human) rather than abort the session. - Expected impact: Prevents single-PR edge cases from aborting the batch.
- Urgency: MEDIUM — infrequent but causes full session abort.
5. Filter or skip self-referential PRs in the candidate pool [MEDIUM]
- What: In
scripts/list-prs.shor the batch loop inscripts/review-batch.sh, add a check to skip PRs whoseheadRepository.nameWithOwnermatchesAGENT_REPO(petry-projects/.github-private) when the change touches CI/review workflow files. - Why: PR fix(pr-review): detect provider usage-cap and fall back to next engine #107 and similar workflow-self-improvement PRs consistently trigger unusual model behaviour (invalid decisions, self-referential confusion) and are reviewed by human developers anyway.
- Expected impact: Avoids model edge-case failures on meta-PRs; reduces unnecessary API calls.
- Urgency: MEDIUM.
6. Upgrade actions/cache to v4-compatible Node.js 24 [LOW]
- What: In
.github/workflows/pr-review.yml, updateactions/cache@v4once a Node.js 24 compatible version is published, or setFORCE_JAVASCRIPT_ACTIONS_TO_NODE24=trueto validate behaviour. Deadline: 2026-06-02 (forced) / 2026-09-16 (Node.js 20 removed). - Why: Every run emits
##[warning]Node.js 20 actions are deprecatedforactions/cache@v4. This will become an error after September 2026. - Expected impact: Eliminates deprecation warnings; prevents future breakage.
- Urgency: LOW — no current impact, deadline is 4 months out.
6. Health Score
Health: 4/10 — Underlying engine is functional but a persistent per-PR session-abort (broodly/pull/30) and silently broken rubber duck leave the workflow running at roughly half capacity with no cross-engine safety net.
Activity