Skip to content

PR Review Agent — failures detected 2026-09-13 #1807

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 998 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 998 1s 44s 21m24s 46m9s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
check_suite 181 55 58 0 295 18%
pull_request 72 14 0 0 86 16%
pull_request_review 26 59 102 0 187 31%
repository_dispatch 124 203 0 45 375 54%
workflow_dispatch 45 6 0 6 57 10%
TOTAL 448 337 160 51 1000 33%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 8
Ticks that re-dispatched a review (hits) 8
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.


PR Review Agent — Health Report

Date: 2026-09-13
Workflow: pr-review-trigger.yml → pr-review.yml@pr-review/v1-next
Repo: petry-projects/.github-private


1. Executive Summary

Status: DEGRADED
Period: 2026-09-12T18:27Z – 2026-09-13T11:39Z (~17.2 hours)
Result: 51 of 499 actionable runs failed (10.2%); 2 additional startup_failure events; 337 cancelled (33.7% of total)

Key findings:

  • A concentrated burst of ~46 failures struck between 05:01–05:50 UTC (49-min window), accounting for ~90% of all failures — the dominant signal
  • A secondary mini-cluster of 4 failures at 06:15 UTC followed immediately after the burst subsided, suggesting the root cause was not fully resolved on first recovery
  • Two startup_failure events at 08:51 and 08:55 UTC are categorically different (no steps executed), indicating a runner/workflow-init issue unrelated to the burst
  • Post-recovery (05:56–11:39 UTC, ~5.7 hours): only 6 scattered failures — the workflow is operationally healthy outside the burst window
  • The 33.7% cancellation rate is consistent with pull_request: synchronize events triggering concurrent runs that GitHub's auto-cancel behavior prunes; this is expected behavior, not a defect

Action required: Pull logs for representative runs from the 05:34–05:50 UTC burst cluster (e.g., gh run view 76052 --log-failed) and for both startup_failure runs (#76185, #76188) to confirm root cause before applying any fix.


2. Failure Breakdown

Failure Category Affected Runs Example Error Message
Transient burst — rate limit or API outage ~46 runs (#75905–#76119, 05:01–05:50 UTC) Log content not captured — see Note below
Secondary post-burst cluster 4 runs (#76129–#76132, 06:15 UTC) Log content not captured
startup_failure — runner / workflow init 2 runs (#76185, #76188, 08:51–08:55 UTC) startup_failure status; no step logs emitted
Isolated / scattered 5 runs (#75314, #75322, #75409, #75496, #75504, prior-day evening) Log content not captured
Isolated post-recovery 1 run (#76161, 08:33 UTC) Log content not captured

Note: The Failed Run Logs section contained no captured output. All categories, root-cause inferences, and urgencies below are derived from run timing, status clustering, workflow structure, and known failure modes for this architecture. Error messages cannot be quoted. Recommendation 1 addresses this gap directly.


3. Error Patterns

Pattern A — Burst cluster (05:01–05:50 UTC, ~46 failures)

Observable signal: ~46 runs failed across a 49-minute window. Failures began sparsely at 05:01 (#75905, #75907, #75912, #75914, #75915), paused briefly, resumed at 05:14 (#75959, #75963, #75968), then intensified sharply from 05:34–05:50 UTC (#76019–#76119, ~38 failures in 16 minutes). Full recovery at 05:56 UTC with no failures in the subsequent batch (#76120–#76128, all success).

Sub-step: The cluster runs were almost all short-duration given the p50 of 44s — failing fast, not timing out. This is consistent with an upstream rejection (rate limit 429, hard auth failure, or broken dependency) rather than a slow resource exhaustion.

Candidate root causes (ranked by fit):

  1. Engine rate limit (Claude/Copilot API quota): The reusable calls an AI review engine. A per-minute or per-hour quota hit would cause all concurrent calls to fail immediately, recover when the window resets, and produce exactly this burst-then-clean-recovery shape. The burst coincides with early-morning UTC, a common peak for US-West repo activity.

  2. GitHub secondary rate limit (429): The review-one-pr.sh script likely issues multiple gh api calls per run. At the volume seen (hundreds of runs/hour), concurrent execution can trigger GitHub's secondary rate limit, producing cascading fast failures.

  3. Bad push to pr-review/v1-next: A defective commit pushed to the next channel at ~05:00 UTC (the ring-0 canary channel) could have broken the reusable, with a fix-forward commit restoring it at ~05:56 UTC. Consistent with the sharp recovery boundary.

  4. Transient infrastructure outage: A brief GitHub Actions backend or network partition to a model-provider endpoint would produce an identical pattern.

Pattern B — Secondary cluster (06:15 UTC, 4 failures: #76129–#76132)

Observable signal: 4 consecutive failures exactly 25 minutes after burst recovery. Surrounded by successes.

Likely cause: Retry flood from cancelled runs that were re-queued once the burst cleared; these re-runs hit a residual rate-limit window or a cooldown state. Alternatively, a second small push event to a heavily-active PR triggered these at an unlucky moment.

Pattern C — startup_failure (#76185, #76188, 08:51–08:55 UTC)

Observable signal: startup_failure status with no step logs — the runner never entered job execution. These occurred ~3 hours after the burst resolved, making them independent.

Candidate root causes:

  1. Required secret temporarily unavailable: A secret referenced via secrets: inherit (e.g., CLAUDE_API_KEY, ANTHROPIC_API_KEY, or a GITHUB_TOKEN scoped secret) was rotated, paused, or scoped out during an ~4-minute window, then restored.
  2. pr-review/v1-next ref briefly unreachable: A tag move via scripts/cut-release.sh creates a window where the called ref is being rewritten. If the tag was force-advanced at 08:51 UTC, the two concurrent callers would hit startup_failure.
  3. Runner capacity exhaustion: No available runners in the pool; unlikely given other runs were succeeding in the same window.

Pattern D — Isolated scattered failures (~6 runs)

Observable signal: Single failures surrounded by successes across the prior-day evening window (#75314 at 18:39, #75322 at 19:08, #75409 at 21:02, #75496 at 22:58, #75504 at 00:49) and one post-burst (#76161 at 08:33).

Likely cause: Individual PR review runs encountering transient API errors (500, 503, timeout) on a single call in review-one-pr.sh or engine.sh without retry handling. These are background noise rather than a systematic issue.


4. Token Scope Analysis

Scopes declared in pr-review-trigger.yml:

Scope Level Purpose
contents read Checkout workflow source; read PR file diffs
pull-requests write Post review comments and approval decisions
checks read Read check suite status for the check_suite: completed trigger
metadata read (implicit) Always granted; required for basic API access

Gap analysis (inferred — no gh auth status output in logs):

Scope Status Assessment
pull-requests: write Declared Sufficient for posting reviews; present
contents: read Declared Sufficient for checkout; present
checks: read Declared Sufficient for check_suite trigger consumption; present
statuses: write Not declared If engine.sh posts commit statuses (e.g., pending/success), this scope is missing and would cause silent 403s
actions: read Not declared If the reusable inspects sibling workflow runs (common in orchestration scripts), this would be missing
Engine API key Via secrets: inherit The startup_failure events suggest this may have been transiently unavailable; verify secret name matches what the reusable expects

Recommendations:

  • Confirm the engine API secret name used in pr-review.yml matches what is present in this repo's secrets store.
  • If engine.sh posts commit statuses, add statuses: write to the permissions block in pr-review-trigger.yml.
  • If post-burst log review reveals 403 errors on any step, cross-reference the failing step's script to identify the missing scope precisely.

5. Recommendations

1. Capture failed-run log output — Root causes for all failure categories are currently inferred, not confirmed.

  • What: Add a log artifact upload step or structured failure summary to pr-review.yml: gh run view ${{ github.run_id }} --log-failed > /tmp/run-failure.log followed by actions/upload-artifact. Alternatively, configure the health-report generator to call gh run view <id> --log-failed for each failed run.
  • Why: The current report has an empty log section, making every diagnosis speculative.
  • Impact: Enables definitive root-cause classification for all future failures within one report cycle.
  • Urgency: CRITICAL

2. Investigate the 05:37–05:50 UTC burst with actual logs — Pull logs for 3–5 representative runs from the peak of the cluster.

  • What: gh run view 76052 --log-failed, gh run view 76075 --log-failed, gh run view 76113 --log-failed (spread across the burst to catch any variation).
  • Why: ~90% of failures originate here. If rate-limit-caused, adding per-run retry backoff in engine.sh or review-one-pr.sh eliminates the dominant failure mode. If a bad next channel push caused it, the tag history will confirm and no code change is needed.
  • Impact: Resolves the dominant failure class entirely if the fix matches the confirmed cause.
  • Urgency: HIGH

3. Investigate startup_failure events (#76185, #76188) — These are categorically different from normal failures and indicate a workflow-init problem.

  • What: gh run view 76185 --log and gh run view 76188 --log. Check whether the failure is in the runner pickup phase or the workflow-ref resolution phase. Cross-reference with any secret rotation events or cut-release.sh tag-move activity logged in the repo around 08:51 UTC.
  • Why: startup_failure is silent — no review is posted, no error is surfaced to PR authors. If caused by a secret rotation window, a race condition with the release process creates a recurring risk.
  • Impact: Eliminates a silent failure mode that is invisible to PR authors.
  • Urgency: HIGH

4. Add exponential-backoff retry in engine.sh / review-one-pr.sh — Conditional on burst investigation confirming rate limiting.

  • What: Wrap AI API and gh api calls in a retry loop, e.g.:
    for attempt in 1 2 3; do
      call_api && break || {
        wait=$(( attempt * 30 + RANDOM % 30 ))
        echo "Attempt $attempt failed; retrying in ${wait}s" >&2
        sleep "$wait"
      }
    done
  • Why: A single retry with 30–90s jitter absorbs transient rate-limit windows (GitHub resets secondary limits in 60s) without failing the run or requiring a re-queue.
  • Impact: Reduces rate-limit-class failure rate to near zero for transient windows under ~90s.
  • Urgency: MEDIUM

5. Add a startup_failure alerting workflow — Real-time visibility when the ring-0 canary goes dark at startup.

  • What: Add a workflow triggered by workflow_run completed with conclusion startup_failure on pr-review-trigger.yml that posts to a notification channel (Slack, GitHub issue, etc.).
  • Why: startup_failure events produce no review output and no visible error to PR authors. Without an alert, ring-0 can be silently non-functional.
  • Impact: Makes ring-0 instability visible in real time; critical for the canary purpose of this repo.
  • Urgency: LOW (infrequent, but the failure mode is silent when it occurs)

6. Review cancellation policy — The 33.7% cancellation rate may be pruning legitimate review attempts during burst pushes.

  • What: Confirm whether pr-review.yml sets concurrency: cancel-in-progress: true at the PR level. If so, evaluate switching to cancel-in-progress: false with a per-PR concurrency group to queue rather than drop runs.
  • Why: During the burst window, cancellations may have silenced review attempts that were valid. With queuing, they would execute once the rate-limit cleared rather than being dropped.
  • Impact: Reduces lost reviews during high-activity windows; no change to pass/fail counts.
  • Urgency: LOW

6. Health Score

Health: 6/10 — Workflow is operationally healthy outside a single 49-minute burst window that produced ~90% of all failures; root cause is unconfirmed due to absent log capture, and two independent startup_failure events add a separate silent failure risk to the ring-0 canary.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions