Repository navigation
Description
Convergence latency (deterministic)
Workflow duration across 998 completed run(s) in the window — computed by pr_review_health.sh, not the model.
| Workflow | Runs | min | p50 | p95 | max |
|---|---|---|---|---|---|
pr-review-trigger.yml |
998 | 1s | 44s | 21m24s | 46m9s |
Run-outcome mix by triggering event
pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).
| Event | success | cancelled | skipped | failure | total | cancel % |
|---|---|---|---|---|---|---|
check_suite |
181 | 55 | 58 | 0 | 295 | 18% |
pull_request |
72 | 14 | 0 | 0 | 86 | 16% |
pull_request_review |
26 | 59 | 102 | 0 | 187 | 31% |
repository_dispatch |
124 | 203 | 0 | 45 | 375 | 54% |
workflow_dispatch |
45 | 6 | 0 | 6 | 57 | 10% |
| TOTAL | 448 | 337 | 160 | 51 | 1000 | 33% |
Sweep hit-rate (deterministic)
How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.
| Metric | Value |
|---|---|
| Scheduled sweep ticks | 8 |
| Ticks that re-dispatched a review (hits) | 8 |
| Sweep hit-rate | 100% |
Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.
PR Review Agent — Health Report
Date: 2026-09-13
Workflow: pr-review-trigger.yml → pr-review.yml@pr-review/v1-next
Repo: petry-projects/.github-private
1. Executive Summary
Status: DEGRADED
Period: 2026-09-12T18:27Z – 2026-09-13T11:39Z (~17.2 hours)
Result: 51 of 499 actionable runs failed (10.2%); 2 additional startup_failure events; 337 cancelled (33.7% of total)
Key findings:
- A concentrated burst of ~46 failures struck between 05:01–05:50 UTC (49-min window), accounting for ~90% of all failures — the dominant signal
- A secondary mini-cluster of 4 failures at 06:15 UTC followed immediately after the burst subsided, suggesting the root cause was not fully resolved on first recovery
- Two
startup_failureevents at 08:51 and 08:55 UTC are categorically different (no steps executed), indicating a runner/workflow-init issue unrelated to the burst - Post-recovery (05:56–11:39 UTC, ~5.7 hours): only 6 scattered failures — the workflow is operationally healthy outside the burst window
- The 33.7% cancellation rate is consistent with
pull_request: synchronizeevents triggering concurrent runs that GitHub's auto-cancel behavior prunes; this is expected behavior, not a defect
Action required: Pull logs for representative runs from the 05:34–05:50 UTC burst cluster (e.g., gh run view 76052 --log-failed) and for both startup_failure runs (#76185, #76188) to confirm root cause before applying any fix.
2. Failure Breakdown
| Failure Category | Affected Runs | Example Error Message |
|---|---|---|
| Transient burst — rate limit or API outage | ~46 runs (#75905–#76119, 05:01–05:50 UTC) | Log content not captured — see Note below |
| Secondary post-burst cluster | 4 runs (#76129–#76132, 06:15 UTC) | Log content not captured |
startup_failure — runner / workflow init |
2 runs (#76185, #76188, 08:51–08:55 UTC) | startup_failure status; no step logs emitted |
| Isolated / scattered | 5 runs (#75314, #75322, #75409, #75496, #75504, prior-day evening) | Log content not captured |
| Isolated post-recovery | 1 run (#76161, 08:33 UTC) | Log content not captured |
Note: The
Failed Run Logssection contained no captured output. All categories, root-cause inferences, and urgencies below are derived from run timing, status clustering, workflow structure, and known failure modes for this architecture. Error messages cannot be quoted. Recommendation 1 addresses this gap directly.
3. Error Patterns
Pattern A — Burst cluster (05:01–05:50 UTC, ~46 failures)
Observable signal: ~46 runs failed across a 49-minute window. Failures began sparsely at 05:01 (#75905, #75907, #75912, #75914, #75915), paused briefly, resumed at 05:14 (#75959, #75963, #75968), then intensified sharply from 05:34–05:50 UTC (#76019–#76119, ~38 failures in 16 minutes). Full recovery at 05:56 UTC with no failures in the subsequent batch (#76120–#76128, all success).
Sub-step: The cluster runs were almost all short-duration given the p50 of 44s — failing fast, not timing out. This is consistent with an upstream rejection (rate limit 429, hard auth failure, or broken dependency) rather than a slow resource exhaustion.
Candidate root causes (ranked by fit):
-
Engine rate limit (Claude/Copilot API quota): The reusable calls an AI review engine. A per-minute or per-hour quota hit would cause all concurrent calls to fail immediately, recover when the window resets, and produce exactly this burst-then-clean-recovery shape. The burst coincides with early-morning UTC, a common peak for US-West repo activity.
-
GitHub secondary rate limit (429): The
review-one-pr.shscript likely issues multiplegh apicalls per run. At the volume seen (hundreds of runs/hour), concurrent execution can trigger GitHub's secondary rate limit, producing cascading fast failures. -
Bad push to
pr-review/v1-next: A defective commit pushed to thenextchannel at ~05:00 UTC (the ring-0 canary channel) could have broken the reusable, with a fix-forward commit restoring it at ~05:56 UTC. Consistent with the sharp recovery boundary. -
Transient infrastructure outage: A brief GitHub Actions backend or network partition to a model-provider endpoint would produce an identical pattern.
Pattern B — Secondary cluster (06:15 UTC, 4 failures: #76129–#76132)
Observable signal: 4 consecutive failures exactly 25 minutes after burst recovery. Surrounded by successes.
Likely cause: Retry flood from cancelled runs that were re-queued once the burst cleared; these re-runs hit a residual rate-limit window or a cooldown state. Alternatively, a second small push event to a heavily-active PR triggered these at an unlucky moment.
Pattern C — startup_failure (#76185, #76188, 08:51–08:55 UTC)
Observable signal: startup_failure status with no step logs — the runner never entered job execution. These occurred ~3 hours after the burst resolved, making them independent.
Candidate root causes:
- Required secret temporarily unavailable: A secret referenced via
secrets: inherit(e.g.,CLAUDE_API_KEY,ANTHROPIC_API_KEY, or aGITHUB_TOKENscoped secret) was rotated, paused, or scoped out during an ~4-minute window, then restored. pr-review/v1-nextref briefly unreachable: A tag move viascripts/cut-release.shcreates a window where the called ref is being rewritten. If the tag was force-advanced at 08:51 UTC, the two concurrent callers would hitstartup_failure.- Runner capacity exhaustion: No available runners in the pool; unlikely given other runs were succeeding in the same window.
Pattern D — Isolated scattered failures (~6 runs)
Observable signal: Single failures surrounded by successes across the prior-day evening window (#75314 at 18:39, #75322 at 19:08, #75409 at 21:02, #75496 at 22:58, #75504 at 00:49) and one post-burst (#76161 at 08:33).
Likely cause: Individual PR review runs encountering transient API errors (500, 503, timeout) on a single call in review-one-pr.sh or engine.sh without retry handling. These are background noise rather than a systematic issue.
4. Token Scope Analysis
Scopes declared in pr-review-trigger.yml:
| Scope | Level | Purpose |
|---|---|---|
contents |
read |
Checkout workflow source; read PR file diffs |
pull-requests |
write |
Post review comments and approval decisions |
checks |
read |
Read check suite status for the check_suite: completed trigger |
metadata |
read (implicit) |
Always granted; required for basic API access |
Gap analysis (inferred — no gh auth status output in logs):
| Scope | Status | Assessment |
|---|---|---|
pull-requests: write |
Declared | Sufficient for posting reviews; present |
contents: read |
Declared | Sufficient for checkout; present |
checks: read |
Declared | Sufficient for check_suite trigger consumption; present |
statuses: write |
Not declared | If engine.sh posts commit statuses (e.g., pending/success), this scope is missing and would cause silent 403s |
actions: read |
Not declared | If the reusable inspects sibling workflow runs (common in orchestration scripts), this would be missing |
| Engine API key | Via secrets: inherit |
The startup_failure events suggest this may have been transiently unavailable; verify secret name matches what the reusable expects |
Recommendations:
- Confirm the engine API secret name used in
pr-review.ymlmatches what is present in this repo's secrets store. - If
engine.shposts commit statuses, addstatuses: writeto thepermissionsblock inpr-review-trigger.yml. - If post-burst log review reveals 403 errors on any step, cross-reference the failing step's script to identify the missing scope precisely.
5. Recommendations
1. Capture failed-run log output — Root causes for all failure categories are currently inferred, not confirmed.
- What: Add a log artifact upload step or structured failure summary to
pr-review.yml:gh run view ${{ github.run_id }} --log-failed > /tmp/run-failure.logfollowed byactions/upload-artifact. Alternatively, configure the health-report generator to callgh run view <id> --log-failedfor each failed run. - Why: The current report has an empty log section, making every diagnosis speculative.
- Impact: Enables definitive root-cause classification for all future failures within one report cycle.
- Urgency: CRITICAL
2. Investigate the 05:37–05:50 UTC burst with actual logs — Pull logs for 3–5 representative runs from the peak of the cluster.
- What:
gh run view 76052 --log-failed,gh run view 76075 --log-failed,gh run view 76113 --log-failed(spread across the burst to catch any variation). - Why: ~90% of failures originate here. If rate-limit-caused, adding per-run retry backoff in
engine.shorreview-one-pr.sheliminates the dominant failure mode. If a badnextchannel push caused it, the tag history will confirm and no code change is needed. - Impact: Resolves the dominant failure class entirely if the fix matches the confirmed cause.
- Urgency: HIGH
3. Investigate startup_failure events (#76185, #76188) — These are categorically different from normal failures and indicate a workflow-init problem.
- What:
gh run view 76185 --logandgh run view 76188 --log. Check whether the failure is in the runner pickup phase or the workflow-ref resolution phase. Cross-reference with any secret rotation events orcut-release.shtag-move activity logged in the repo around 08:51 UTC. - Why:
startup_failureis silent — no review is posted, no error is surfaced to PR authors. If caused by a secret rotation window, a race condition with the release process creates a recurring risk. - Impact: Eliminates a silent failure mode that is invisible to PR authors.
- Urgency: HIGH
4. Add exponential-backoff retry in engine.sh / review-one-pr.sh — Conditional on burst investigation confirming rate limiting.
- What: Wrap AI API and
gh apicalls in a retry loop, e.g.:for attempt in 1 2 3; do call_api && break || { wait=$(( attempt * 30 + RANDOM % 30 )) echo "Attempt $attempt failed; retrying in ${wait}s" >&2 sleep "$wait" } done
- Why: A single retry with 30–90s jitter absorbs transient rate-limit windows (GitHub resets secondary limits in 60s) without failing the run or requiring a re-queue.
- Impact: Reduces rate-limit-class failure rate to near zero for transient windows under ~90s.
- Urgency: MEDIUM
5. Add a startup_failure alerting workflow — Real-time visibility when the ring-0 canary goes dark at startup.
- What: Add a workflow triggered by
workflow_runcompleted with conclusionstartup_failureonpr-review-trigger.ymlthat posts to a notification channel (Slack, GitHub issue, etc.). - Why:
startup_failureevents produce no review output and no visible error to PR authors. Without an alert, ring-0 can be silently non-functional. - Impact: Makes ring-0 instability visible in real time; critical for the canary purpose of this repo.
- Urgency: LOW (infrequent, but the failure mode is silent when it occurs)
6. Review cancellation policy — The 33.7% cancellation rate may be pruning legitimate review attempts during burst pushes.
- What: Confirm whether
pr-review.ymlsetsconcurrency: cancel-in-progress: trueat the PR level. If so, evaluate switching tocancel-in-progress: falsewith a per-PR concurrency group to queue rather than drop runs. - Why: During the burst window, cancellations may have silenced review attempts that were valid. With queuing, they would execute once the rate-limit cleared rather than being dropped.
- Impact: Reduces lost reviews during high-activity windows; no change to pass/fail counts.
- Urgency: LOW
6. Health Score
Health: 6/10 — Workflow is operationally healthy outside a single 49-minute burst window that produced ~90% of all failures; root cause is unconfirmed due to absent log capture, and two independent startup_failure events add a separate silent failure risk to the ring-0 canary.
Activity