You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Fault-isolate the catch-up sweep so a single transient GitHub API error on one PR no longer aborts the entire scheduled run (the source of the 11.1% failure rate).
Root Cause
sweep-dispatch.sh runs under set -euo pipefail. The repository_dispatch POST (gh api --method POST .../dispatches) inside the candidate loop is unguarded and executes with errexit active β a single transient failure (secondary rate limit / 5xx / network blip) aborts the whole run, fails the workflow, and strands the remaining candidates. The gh pr view / gh api graphql fact-gathering calls in evaluate_pr can also transiently fail and yield silently-wrong decisions. The sweep is idempotent and runs every 15 min, so a transient per-PR failure should be logged and retried next cycle, not fail the run.
Implementation Checklist
.github/scripts/pr-auto-review/sweep-dispatch.sh β add a bounded-retry dispatch_review helper; on persistent failure log ::warning:: and continue (do not abort). Track/report failed dispatches.
.github/scripts/pr-auto-review/sweep-dispatch.sh β harden evaluate_pr fact-gathering: guard gh pr view / gh api graphql; on hard fetch failure emit a skip-error class and skip the PR this cycle instead of dispatching on garbage.
.github/scripts/pr-auto-review/README.md β document the resilience behaviour.
Tests to Write
test/workflows/pr-auto-review/sweep-dispatch.bats β a transient dispatch failure on one PR does not abort the sweep; the other PR still dispatches; run exits 0; warning logged.
test/workflows/pr-auto-review/sweep-dispatch.bats β a transient gh pr view failure on one PR is skipped (skip-error) and the loop continues.
test/workflows/pr-auto-review/sweep-dispatch.bats β a dispatch that fails once then succeeds is retried (no lost dispatch, no failure).
Test Command
bats --print-output-on-failure test/workflows/pr-auto-review/ plus shellcheck -x .github/scripts/pr-auto-review/**/*.sh and bash .dev-lead/scripts/dev-lead-lint.sh
Root-caused the 11.1% failure rate: sweep-dispatch.sh runs under set -euo pipefail; the repository_dispatch POST inside the candidate loop was unguarded and ran with errexit active, so a single transient API error (secondary rate limit / 5xx / network blip) on one PR aborted the whole scheduled run and stranded the remaining candidates.
Added a bounded-retry dispatch_review helper; on persistent failure it logs a ::warning:: and the loop continues (the run still exits 0, since the sweep is idempotent and re-runs every 15 min).
Hardened evaluate_pr fact-gathering (gh pr view, review-threads GraphQL): a transient fetch failure now yields a skip-error class and skips that PR this cycle instead of dispatching on incomplete facts.
Documented the fault-isolation behaviour in the pr-auto-review README.
Wrote failing end-to-end tests first (TDD), then implemented until green.
Test Results
=== bats (test/workflows/pr-auto-review/) ===
ok 74 candidates: empty stdin emits nothing
ok 75 candidates: non-array API error body emits nothing
ok 76 candidates: an object missing isDraft is treated as non-draft
ok 77 candidates: all drafts β nothing
77 tests, 0 failures (73 pre-existing + 4 new sweep-dispatch resilience tests)
=== new sweep-dispatch.bats ===
ok 1 sweep: both ready PRs dispatch and the run succeeds
ok 2 sweep: a transient dispatch failure on one PR does not abort the run
ok 3 sweep: a transient 'gh pr view' failure skips that PR and continues
ok 4 sweep: a dispatch that fails once is retried and then succeeds
=== shellcheck -x (ready-check.sh, sweep.sh, sweep-dispatch.sh) ===
shellcheck OK
=== .dev-lead/scripts/dev-lead-lint.sh ===
[lint] all checks passed
Files Changed
.github/scripts/pr-auto-review/sweep-dispatch.sh: bounded-retry dispatch_review + per-PR fault isolation so one transient error no longer aborts the sweep; new DISPATCH_RETRIES / DISPATCH_RETRY_SLEEP knobs; failed-dispatch count in the summary line.
test/workflows/pr-auto-review/sweep-dispatch.bats: new end-to-end resilience suite driving the orchestrator with a scripted gh stub.
This targets the failure-rate threshold that tripped the monitor (>10%). The p95 duration tail (~17456s) is queue-wait from the serialized concurrency: pr-auto-review-sweep group (cancel-in-progress: false) on the 15-min schedule, not per-run compute β each run is still bounded by timeout-minutes: 20. Reducing aborted runs also removes the retry/backlog pressure that lengthens that queue.
β Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-08-07 (3 days) β the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.
Activity
don-petry commentedon Aug 7, 2026
Dev-Lead Implementation Plan
Issue: #947 β [Fleet Monitor] petry-projects/.github β .github/workflows/pr-auto-review-sweep.yml
Scope
Fault-isolate the catch-up sweep so a single transient GitHub API error on one PR no longer aborts the entire scheduled run (the source of the 11.1% failure rate).
Root Cause
sweep-dispatch.shruns underset -euo pipefail. Therepository_dispatchPOST (gh api --method POST .../dispatches) inside the candidate loop is unguarded and executes with errexit active β a single transient failure (secondary rate limit / 5xx / network blip) aborts the whole run, fails the workflow, and strands the remaining candidates. Thegh pr view/gh api graphqlfact-gathering calls inevaluate_prcan also transiently fail and yield silently-wrong decisions. The sweep is idempotent and runs every 15 min, so a transient per-PR failure should be logged and retried next cycle, not fail the run.Implementation Checklist
.github/scripts/pr-auto-review/sweep-dispatch.shβ add a bounded-retrydispatch_reviewhelper; on persistent failure log::warning::and continue (do not abort). Track/report failed dispatches..github/scripts/pr-auto-review/sweep-dispatch.shβ hardenevaluate_prfact-gathering: guardgh pr view/gh api graphql; on hard fetch failure emit askip-errorclass and skip the PR this cycle instead of dispatching on garbage..github/scripts/pr-auto-review/README.mdβ document the resilience behaviour.Tests to Write
test/workflows/pr-auto-review/sweep-dispatch.batsβ a transient dispatch failure on one PR does not abort the sweep; the other PR still dispatches; run exits 0; warning logged.test/workflows/pr-auto-review/sweep-dispatch.batsβ a transientgh pr viewfailure on one PR is skipped (skip-error) and the loop continues.test/workflows/pr-auto-review/sweep-dispatch.batsβ a dispatch that fails once then succeeds is retried (no lost dispatch, no failure).Test Command
bats --print-output-on-failure test/workflows/pr-auto-review/plusshellcheck -x .github/scripts/pr-auto-review/**/*.shandbash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Aug 7, 2026
Dev-Lead: Implementation Complete
Plan Execution
sweep-dispatch.shruns underset -euo pipefail; therepository_dispatchPOST inside the candidate loop was unguarded and ran with errexit active, so a single transient API error (secondary rate limit / 5xx / network blip) on one PR aborted the whole scheduled run and stranded the remaining candidates.dispatch_reviewhelper; on persistent failure it logs a::warning::and the loop continues (the run still exits 0, since the sweep is idempotent and re-runs every 15 min).evaluate_prfact-gathering (gh pr view, review-threads GraphQL): a transient fetch failure now yields askip-errorclass and skips that PR this cycle instead of dispatching on incomplete facts.Test Results
Files Changed
.github/scripts/pr-auto-review/sweep-dispatch.sh: bounded-retrydispatch_review+ per-PR fault isolation so one transient error no longer aborts the sweep; newDISPATCH_RETRIES/DISPATCH_RETRY_SLEEPknobs; failed-dispatch count in the summary line.test/workflows/pr-auto-review/sweep-dispatch.bats: new end-to-end resilience suite driving the orchestrator with a scriptedghstub..github/scripts/pr-auto-review/README.md: documents the per-PR fault-isolation behaviour ([Fleet Monitor] petry-projects/.github β .github/workflows/pr-auto-review-sweep.ymlΒ #947).Notes
concurrency: pr-auto-review-sweepgroup (cancel-in-progress: false) on the 15-min schedule, not per-run compute β each run is still bounded bytimeout-minutes: 20. Reducing aborted runs also removes the retry/backlog pressure that lengthens that queue.pr_auto_review_ready/sweep.sh); the pr-auto-review ready-check counts non-required/cancelled checks β false 'not passing', blocks auto-dispatchΒ #680 required-vs-non-required and dev-lead fix-review addresses advisory findings in code but never resolves the review thread β PRs stall on the advisory gateΒ #806/dev-lead fix-review won't act on advisory-bot nitpick threads (green CI + CHANGES_REQUESTED, empty machine findings β status=no-changes β PR stalls on human)Β #892 thread tolerances are untouched.skip-erroris an orchestrator-level log class only; it does not alter the documentedpr_auto_review_readyclass set.don-petry commentedon Aug 10, 2026
β Auto-resolved by Fleet Monitor. This workflow's failure rate has been below the 10% threshold (or the workflow was removed) since 2026-08-07 (3 days) β the monitor stopped flagging it. Closing. It will reopen automatically if the workflow crosses the threshold again.
feat: implement issue #947 β [Fleet Monitor] petry-projects/.github ββ¦