Skip to content

PR Review Agent — failures detected 2026-09-16 #1823

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 168 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 168 2s 37s 12m12s 17m43s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
repository_dispatch 61 37 0 9 107 34%
workflow_dispatch 60 0 0 1 61 0%
TOTAL 121 37 0 10 168 22%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 6
Ticks that re-dispatched a review (hits) 6
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.


PR Review Agent — Health Report

Workflow: pr-review-trigger.yml | Repo: petry-projects/.github-private
Generated: 2026-09-16


1. Executive Summary

Status: DEGRADED
Period: 2026-09-15T11:49Z – 2026-09-16T10:00Z (≈22 h)
Result: 10 of 168 runs failed (5.95%) | 37 cancelled (22%) | 121 succeeded (72%)

Key findings:

  • Failed run logs were not captured / returned empty — root cause cannot be confirmed from log text alone
  • Failures are temporally clustered: 2 at 11:49, 1 at 12:12, 4 in 17:16–17:55, 2 at 20:26, 1 at 01:06 — suggests a recurring transient condition rather than a persistent broken state
  • Cancellations (22%) are burst-correlated (e.g. 21:28–21:43 window: 7 cancels around 3 successes) — consistent with expected cancel-in-progress concurrency group behavior, not a fault signal
  • p95 duration 12m12s vs p50 37s indicates a bimodal distribution: most runs are fast no-ops (no eligible PR); long-tail runs are real reviews

Action required: Re-run log capture with step-level detail enabled (--log flag or gh run view --log) for any future failure so root cause can be confirmed; current empty log payload blocks diagnosis.


2. Failure Breakdown

Failure Category Affected Runs Example Error Message
Unknown — logs not captured 10 (#77148, #77149, #77151, #77168, #77170, #77176, #77182, #77218, #77219, #77287) (no log output returned in report payload)
Transient / infrastructure (inferred from clustering) Likely subset of above (unconfirmed — see §3)

Note: The ## Failed Run Logs section in the source report payload contained no content (only a horizontal rule). All failure analysis below is inferred from timing patterns and workflow structure. The categories above cannot be confirmed without log text.


3. Error Patterns

Pattern A — Back-to-back failure pairs
Runs #77148/#77149 (11:49:26Z / 11:49:27Z) and #77218/#77219 (20:26:14Z / 20:26:16Z) failed within 1–2 seconds of each other. This is characteristic of either:

  • A shared upstream event (e.g. a push to pr-review/v1-next tag) triggering multiple runs simultaneously and both hitting the same broken state
  • A GitHub API or rate-limit error that is instantaneous (no retry window)

Because these pairs resolve without further failures in the immediate next run (e.g. #77220 succeeds at 20:26:46Z), the condition is transient.

Pattern B — Rolling failures in the 17:16–17:55 window
Four failures across 39 minutes (#77168, #77170, #77176, #77182), each separated by a successful run (#77167, #77169, #77174, #77180). This alternating success/failure pattern under the same workflow version suggests:

  • Per-PR state (e.g. a specific PR causing the agent to error out consistently) rather than a global outage
  • OR a flaky external dependency (Claude API quota, GitHub secondary rate limit) that clears between retries

Pattern C — Isolated single failure at 01:06
Run #77287 is a lone failure preceded by successes and followed by successes. Likely a transient infrastructure or API hiccup.

Root cause cannot be confirmed without log content. The most probable candidates given the clustering behavior are: (1) per-PR agent errors (unhandled edge case in review-one-pr.sh), (2) transient GitHub API secondary rate limiting, or (3) a brief Claude API quota event during the 17:xx burst.


4. Token Scope Analysis

Scopes granted by pr-review-trigger.yml:

Scope Level
contents: read Repository
pull-requests: write Repository
checks: read Repository
secrets: inherit Passed through to called workflow

Assessment:

Scope Status Notes
contents: read Present Required for script checkout at agent_ref
pull-requests: write Present Required to post reviews/comments
checks: read Present Required for check_suite trigger gating
actions: read Not granted May be needed if pr-review.yml reads workflow run status internally
statuses: read Not granted Needed if the agent reads commit statuses to gate reviews
members: read (org-level) Not grantable here If the agent checks org membership for maintainer detection, requires a PAT with read:org — GITHUB_TOKEN cannot grant this

Recommendation: Without log confirmation, the missing read:org scope is the highest-probability invisible failure mode for maintainer-gating logic (consistent with issue #1816 context in recent commits). If review-one-pr.sh calls gh api orgs/{org}/members and gets a 404/403, the run fails silently.


5. Recommendations

  1. [CRITICAL] Capture step-level logs for failed runs in the health report script

    • What: In the script that generates this report, replace gh run list result-only fetching with gh run view <id> --log (or gh run view <id> --log-failed) for any run where conclusion == failure
    • Why: The current report payload has an empty ## Failed Run Logs section — without log text, no failure can be diagnosed or verified
    • Expected impact: Immediate visibility into every future failure; eliminates the "unknown" category entirely
    • Urgency: CRITICAL
  2. [HIGH] Investigate whether read:org scope is needed for maintainer detection

    • What: In scripts/review-one-pr.sh (or engine.sh), search for any call to gh api orgs/ or org membership endpoints; if present, ensure the workflow uses a PAT with read:org rather than GITHUB_TOKEN
    • Why: Recent commit e37e9eb fixed maintainer-comment detection issues; if the fix relies on org API calls, GITHUB_TOKEN will silently 404 on private org membership queries — this would explain the 17:xx alternating-failure pattern where some PRs trigger the org check and others don't
    • Expected impact: Eliminates any org-API-driven failures
    • Urgency: HIGH
  3. [MEDIUM] Investigate the 17:16–17:55 alternating success/failure pattern for per-PR correlation

    • What: Cross-reference which PR URLs were under review during runs #77168, #77170, #77176, #77182 by checking gh run view <id> --json for the triggering event payload; determine if the same PR number appears in all four failures
    • Why: If a single open PR is consistently causing failures, it may be an edge case in review-one-pr.sh (malformed diff, unusual branch name, very large changeset) that needs a guard
    • Expected impact: Narrows root cause to a specific PR or code path
    • Urgency: MEDIUM
  4. [MEDIUM] Add concurrency group documentation / confirm cancel-in-progress is intentional

    • What: Verify pr-review.yml (called workflow at pr-review/v1-next) includes concurrency: cancel-in-progress: true; if so, add a comment in pr-review-trigger.yml noting that the 20–25% cancellation rate is expected
    • Why: The 37 cancellations (22%) will continue to trigger alert fatigue if not documented as intentional; they are not a failure signal but appear alarming in aggregate
    • Expected impact: Cleaner signal-to-noise in future health reports
    • Urgency: MEDIUM
  5. [LOW] Add a workflow_run or schedule-based health check that re-runs failed jobs automatically

    • What: Consider a short retry (1 attempt) on transient failures via if: failure() step that re-dispatches via gh workflow run
    • Why: Pattern C (isolated single failure at 01:06) and Pattern A (2-second back-to-back failures that self-resolve) are likely retriable transients that currently require manual re-run
    • Expected impact: Reduces effective failure rate for transient causes without masking real bugs
    • Urgency: LOW

6. Health Score

Health: 6/10 — Majority of runs succeed but 10 failures are entirely undiagnosed due to missing log capture, making the true severity unknown and the system unmonitorable.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions