Skip to content

PR Review Agent — failures detected 2026-09-30 #1984

Closed

Description

@github-actions

Convergence latency (deterministic)

Workflow duration across 700 completed run(s) in the window — computed by pr_review_health.sh, not the model.

Workflow Runs min p50 p95 max
pr-review-trigger.yml 700 1s 44s 14m39s 22m54s

Run-outcome mix by triggering event

pr-review run conclusions over the window, split by the event that triggered each run — computed deterministically by pr-review-outcomes.sh, not the model. A high cancelled share means runs were evicted while PENDING (concurrency admits one running + one pending per group); a cancelled run is non-blocking but historically rendered as a failing check (#1421).

Event success cancelled skipped failure total cancel %
check_suite 88 143 18 0 249 57%
pull_request 52 0 0 0 52 0%
pull_request_review 12 17 44 0 73 23%
repository_dispatch 83 100 0 4 187 53%
workflow_dispatch 126 11 0 2 139 7%
TOTAL 361 271 62 6 700 38%

Sweep hit-rate (deterministic)

How often the pr-review-sweep scheduled backstop actually had to re-review a PR — computed deterministically by pr-review-sweep-metrics.sh, not the model. The scheduled (cron) path re-dispatches only for genuinely un-eventable cases (GitHub-App / cross-repo checks like SonarCloud that emit no workflow_run); the workflow_run fast path handles every eventable case, so a scheduled re-dispatch is always a case the fast path did not cover.

Metric Value
Scheduled sweep ticks 5
Ticks that re-dispatched a review (hits) 5
Sweep hit-rate 100%

Interpretation: a low rate confirms the timer stays exception-only (the event fast path is carrying the load); a rising rate is a signal that an eventable case is leaking onto the timer — a hole in the narrowing worth investigating.

Now I have all the data needed. Writing the health report.


PR Review Agent — Health Report

Workflow: pr-review-trigger.yml · Repo: petry-projects/.github-private
Generated: 2026-09-30 · Window: 2026-09-29 → 2026-09-30 (last 24 h)


1. Executive Summary

Status: WARNING
Period: 2026-09-29T13:22Z → 2026-09-30T11:21Z
Result: 6 of 700 runs failed (0.86%)

Key findings:

  • All 6 failures are concentrated in repository_dispatch (4) and workflow_dispatch (2) — event-driven and mention-triggered paths only; check_suite, pull_request, and pull_request_review show zero failures
  • Scheduled sweep hit-rate is 100% (5/5 ticks re-dispatched) — every cron tick is catching un-reviewed PRs, indicating a gap in the event fast-path for at least some eventable cases
  • pull_request_review skips 60% of triggers (44/73) — by-design filter logic, but the ratio is high enough to re-audit whether legitimate review-state changes are being dropped
  • No error log content was captured for the 6 failed runs; root cause cannot be confirmed from logs alone

Action required: Retrieve and inspect logs for failed runs #85083, #85089, #85490, #85528, #85636, #85680 to confirm root cause; separately investigate why the scheduled sweep is catching PRs on every tick rather than acting as an exception backstop.


2. Failure Breakdown

Failure Category Affected Runs Example Error Message
repository_dispatch path failure 4 runs (#85490, #85528, #85636, #85680) (logs not captured — no output in failed run logs section)
workflow_dispatch path failure 2 runs (#85083, #85089) (logs not captured — no output in failed run logs section)
check_suite / pull_request / pull_request_review 0 —

Note: The ## Failed Run Logs section provided to this analysis contained no log output. All root-cause analysis below is therefore based on statistical patterns and workflow source inspection only. Definitive error messages must be retrieved via gh run view <id> --log-failed.


3. Error Patterns

Pattern A — repository_dispatch failures (4 runs, 2.1% of that event type)

What is known: The repository_dispatch path is triggered by the mention-listener in petry-projects/.github via pr-review-mention.yml → org reusable → repository_dispatch. The payload carries { pr_url, force_review }.

Plausible causes (ranked by likelihood, not confirmed):

  1. Rate limit or quota exhaustion — The repository_dispatch cluster at ~18:44–19:07 on 2026-09-29 shows ~80 runs in ~23 minutes, many cancelled by concurrency eviction. If the Claude/engine tier was saturated, reviews that made it past the concurrency gate could time out or receive a quota error from the agent engine.

  2. Token or permission error on the dispatch payload — The mention-listener calls back into .github-private via repository_dispatch. If the PAT or GitHub App token used for that cross-repo dispatch lacks repo scope on .github-private, the reusable's secrets: inherit chain may resolve a null secret that then fails an API call inside engine.sh.

  3. Stale pr_url in payload — A force-review dispatch against a PR that was merged or closed between mention and execution would cause review-one-pr.sh to fail on the gh pr view call.

Pattern B — workflow_dispatch failures (2 runs, 1.4% of that event type)

What is known: Both failures (#85083 at 13:22Z, #85089 at 15:56Z) occur about 2.5 hours apart early in the window, before the high-volume burst at 18:44Z.

Plausible causes:

  1. Manual invocation with a bad pr_url — workflow_dispatch accepts an optional pr_url input. An invalid URL (wrong repo, closed PR, typo) would fail inside the reusable's PR enumeration step.

  2. agent_ref: pr-review/v1-next not resolving — If a release was being cut around 13:00–16:00Z and the next tag was temporarily detached, the uses: …@pr-review/v1-next call would fail at checkout with a ref-not-found error.

Pattern C — Sweep 100% hit-rate

This is not a failure, but is a WARNING signal. The scheduled backstop is designed for non-eventable cases (cross-repo checks like SonarCloud that don't emit workflow_run). A 100% hit-rate over 5 ticks means either:

  • There is consistently at least one open PR where the event fast-path did not fire, OR
  • The fast-path is missing a trigger (e.g. the check_suite narrowing condition excludes a check context that should be eventable)

4. Token Scope Analysis

Scopes granted in workflow source (pr-review-trigger.yml):

Permission Level Used for
contents: read job-level (passed to reusable) Checkout of scripts at agent_ref
pull-requests: write job-level (passed to reusable) Posting review comments
checks: read job-level (passed to reusable) Reading check suite conclusions

Scopes not explicitly listed but required by the reusable's known operations:

Scope Status Evidence
issues: write Unknown — not in this stub engine.sh may post issue comments on @mention flow
statuses: read or checks: read Present Needed to gate on CI state before reviewing
actions: read Not granted in stub Reusable may need it to read its own run context; if absent, gh run view calls inside the engine fail silently

Recommendation:

  • actions: read: Add to the stub's permissions block if the reusable's engine.sh calls gh run list or gh run view to inspect check context. Absence would produce a 403 that surfaces as a generic failure.
  • Cross-repo dispatch token: Confirm the secret forwarded via secrets: inherit for the repository_dispatch path has repo scope on .github-private. The mention-listener lives in .github (public repo context); its token may not automatically have write access to the private repo.

5. Recommendations

  1. Retrieve failed run logs immediately

    for id in 85083 85089 85490 85528 85636 85680; do
      gh run view $id --log-failed --repo petry-projects/.github-private
    done
    

    Why: All failure analysis here is probabilistic; log output will confirm or eliminate each hypothesis in minutes.
    Impact: Unblocks definitive root cause and prevents recurrence.
    Urgency: [HIGH]

  2. Investigate 100% sweep hit-rate
    Audit pr-review-sweep-metrics.sh output and identify which PRs were caught exclusively by the cron path. Check whether their triggering check contexts appear in the check_suite.check_suite.app.slug narrowing condition inside pr-review.yml.
    Why: The event fast-path should carry virtually all load; the timer firing every tick means either a missing narrowing condition or a consistent class of PR that escapes event coverage.
    Impact: Reducing sweep reliance lowers latency for affected PRs (cron fires at most every 15–30 min vs. near-realtime on event).
    Urgency: [MEDIUM]

  3. Audit pull_request_review skip rate
    44 of 73 pull_request_review events are skipped (60%). Confirm the skip condition in pr-review.yml (likely review.state != 'approved' && review.state != 'changes_requested') is not too narrow. If "commented" reviews from bots trigger the event but are intentionally skipped, document and confirm. If legitimate human reviews are being dropped, widen the condition.
    Why: A missed changes_requested event means the agent never re-reviews after a blocking review is left.
    Impact: Restores review coverage for filtered event types.
    Urgency: [MEDIUM]

  4. Add actions: read to pr-review-trigger.yml permissions if confirmed missing

    permissions:
      contents: read
      pull-requests: write
      checks: read
      actions: read   # add if engine.sh uses gh run view

    File: .github/workflows/pr-review-trigger.yml, permissions block (line ~46).
    Why: Without it, any gh run call inside the reusable produces a 403 that fails the job.
    Impact: Eliminates one class of silent permission failure.
    Urgency: [MEDIUM] — confirm from logs first (test issue from agent #1 above).

  5. Verify the cross-repo dispatch token scope for @mention path
    Confirm the secret used by pr-review-mention.yml to fire repository_dispatch into .github-private has repo scope (or at minimum write:repo_hook). Check the App installation scopes if it uses a GitHub App.
    Why: 2.1% failure rate on repository_dispatch vs. 1.4% on workflow_dispatch suggests the mention-triggered path carries extra risk from the cross-repo token boundary.
    Impact: Eliminates auth-related failures on @mention-triggered reviews.
    Urgency: [MEDIUM]

  6. Consider a concurrency-eviction label or conclusion override
    The note in the repo (feat: implement issue #1405 — [Phase 2] Add the "How to add an agentic role" runbook and a PR-template interaction-contract section #1421) observes that cancelled (evicted) runs render as failing checks. With a 57% cancellation rate on check_suite events, this could produce noisy red checks on PRs. If not already addressed by feat: implement issue #1405 — [Phase 2] Add the "How to add an agentic role" runbook and a PR-template interaction-contract section #1421, add a concurrency.cancel-in-progress: false or a post-job step that re-sets the check conclusion.
    Why: Cancelled-renders-as-failed degrades PR author experience and may mask real failures.
    Impact: Reduces false-positive failing checks.
    Urgency: [LOW] (cosmetic/UX, not blocking)


6. Health Score

Health: 8/10 — Failure rate is low (0.86%) and the hot path is clean, but log-less failures on the mention/dispatch path and a 100% sweep hit-rate signal two issues that need investigation before they widen.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    automated-reportCreated by automated workflowhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions