Skip to content

PR Review Agent — failures detected 2026-08-20 #1560

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 821 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 821 1s 44s 10m22s 77m12s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
check_suite 123 56 30 0 209 26%
pull_request 37 9 0 3 49 18%
pull_request_review 48 76 43 3 170 44%
repository_dispatch 120 124 0 2 246 50%
workflow_dispatch 145 8 0 0 153 5%
TOTAL 473 273 73 8 827 33%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 35
Ticks that re-dispatched a review (hits) 35
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.


PR Review Agent — Health Report

Workflow: pr-review-trigger.yml | Repo: petry-projects/.github-private
Generated: 2026-08-20 | Window: 2026-08-19 → 2026-08-20


1. Executive Summary

Status: WARNING
Period: 2026-08-19T07:22Z → 2026-08-20T06:57Z (~24 h)
Result: 8 of 754 completed runs failed (1.1%) — excluding 73 skipped

Key findings:

  • All 8 failures occurred in three tight temporal clusters (≤11 min each), indicating transient systemic events rather than persistent misconfiguration
  • No failed-run log content was returned by the log fetcher — root causes cannot be confirmed from logs; assessment is pattern-inference only
  • 273 of 827 runs cancelled (33%) — elevated but consistent with concurrency-group churn during high-event-volume windows (02:40–03:20 UTC burst shows ~80 cancellations in 40 min)

Action required: Retrieve and inspect logs for runs #66781, #66784, #66786, #66787 (the 4-failure burst at 19:12–19:13 UTC) to confirm root cause before the pattern recurs.


2. Failure Breakdown

Failure Category Affected Runs Example Error Message
Transient cluster — 4 rapid failures #66781, #66784, #66786, #66787 (19:12–19:13 UTC) (log content not retrieved)
Transient cluster — 2 failures #66746, #66754 (16:04–16:15 UTC) (log content not retrieved)
Transient cluster — 2 failures #66886, #66937 (02:27–02:33 UTC) (log content not retrieved)
Unknown / unclassified All 8 Log fetch returned no content for failed runs

Note: The ## Failed Run Logs section contained no log content. All category assignments below are inferred from timing patterns. The recommendations section includes steps to retrieve the actual logs.


3. Error Patterns

Pattern A — 4-failure burst (2026-08-19 19:12–19:13 UTC)

  • Runs: #66781, #66784, #66786, #66787
  • Duration: ~2 minutes, 4 consecutive failures
  • Context: Surrounded by cancellations (#66779, #66778, #66782, #66783, #66785) and then immediate recovery (#66789, #66790 succeed at 19:14 UTC)
  • Likely cause: A hard failure in the called reusable at pr-review.yml@pr-review/v1-next — either the reusable itself errored (job-level failure, not step-level), a GitHub Actions infrastructure blip, or a brief token/API outage. The 2-minute total window and immediate recovery argue against a configuration regression.

Pattern B — 2-failure clusters (16:04–16:15 UTC and 02:27–02:33 UTC)

  • Runs: #66746, #66754 and #66886, #66937
  • Duration: 11 min and 6 min respectively
  • Context: Each cluster is preceded and followed by successful runs; cancellations are interleaved (normal concurrency behavior).
  • Likely cause: Same class as Pattern A — transient infrastructure or upstream API errors. The reusable call across the job boundary means the outer trigger job can fail with no step-level log if the inner workflow errors before producing output.

Pattern C — High cancellation volume (02:40–03:20 UTC)

  • Runs: ~80 cancellations in 40 minutes
  • Cause: Expected behavior. A high-volume push or rebase event flooded the trigger queue; the concurrency group cancelled superseded runs. This is not a failure mode — it is the intended behavior of the concurrency strategy.

4. Token Scope Analysis

Scopes granted by the trigger stub

From the workflow source:

permissions:
  contents: read
  pull-requests: write
  checks: read

These are passed to the called reusable (pr-review.yml@pr-review/v1-next) via secrets: inherit.

Scope Status Notes
contents: read Present Required for repo checkout and reading files
pull-requests: write Present Required for posting review comments
checks: read Present Required for reading check suite status
statuses: read Not granted May be needed if the engine reads commit status in addition to checks
actions: read Not granted May be needed if the reusable inspects other workflow runs

Assessment

No permission errors are visible in logs (logs were not returned). The scope set matches the declared minimum. If failures are permission-related, the most likely gap is statuses: read (commit statuses vs. check runs are separate APIs). This cannot be confirmed without log content.

Recommendation: If log retrieval reveals 403 errors, add statuses: read to the permissions block. If the reusable calls gh api /repos/.../actions/runs, also add actions: read.


5. Recommendations

1. Retrieve failed-run logs immediately [HIGH]

  • What: Run gh run view <id> --log-failed for #66781, #66784, #66786, #66787 (the 4-run burst) and #66746, #66754
  • Why: All root-cause analysis in this report is inferred from timing. The actual error class (auth, rate limit, engine quota, infra) is unknown.
  • Expected impact: Converts this WARNING report to a definitive diagnosis.
  • Urgency: HIGH — the 4-failure burst is the largest single event; if it was a rate limit or token error it may recur.

2. Instrument the log fetcher to capture failed-run output [HIGH]

  • What: The health-check script that produced this report returned no content for failed runs. Verify that gh run view --log-failed is being called (not just --log) and that the output is piped into the report.
  • Why: A health report with empty failure logs cannot diagnose root causes. This is a gap in observability.
  • Expected impact: Future reports will include actual error messages.
  • Urgency: HIGH — the monitoring tool is partially blind.

3. Add a concurrency guard with a fail-fast timeout [MEDIUM]

  • What: In pr-review-trigger.yml, add a timeout-minutes to the review job (e.g., timeout-minutes: 20) matching or slightly above the p95 duration (10m22s observed; 20 min gives margin).
  • Why: Without a timeout, a hung run in the called reusable holds a concurrency slot indefinitely. The 77m12s max duration is an outlier worth bounding.
  • Expected impact: Prevents slot starvation during outages; reduces blast radius of stuck runs.
  • Urgency: MEDIUM.

4. Alert on cluster failures (≥3 failures within 5 minutes) [MEDIUM]

  • What: Add a step in the health-check workflow that fires a notification (GitHub issue, Slack, etc.) when failure count exceeds a threshold within a rolling window.
  • Why: The 19:12–19:13 UTC burst (4 failures in 2 min) was self-recovering but went unalerted. If the reusable at pr-review/v1-next is broken, the signal is 4 rapid failures — not 1.
  • Expected impact: Faster human response to systemic events.
  • Urgency: MEDIUM.

5. Investigate the 02:40–03:20 UTC cancellation storm [LOW]

  • What: Identify what event caused ~80 workflow triggers in 40 minutes (mass-push, rebase flood, or external dispatch).
  • Why: The cancellation volume is not harmful but is noisy; if triggered by an automation loop it could mask real failures.
  • Expected impact: Reduced noise in run history; cheaper GitHub Actions minutes.
  • Urgency: LOW.

6. Health Score

Health: 7/10 — Failure rate is low (1.1%) and self-recovering, but undiagnosed root causes and a blind log fetcher leave the system flying partially instrumented.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions