Skip to content

P2 (#860 follow-up): runaway-PR detection in daily health check (commits/comments/cycles/age thresholds) #948

Closed

Description

@don-petry

Context

Part of the #860 runaway remediation (tracker: #926). The per-PR budget (#928) and the org self-trigger fix (combsmith/.github#538) prevent/bound runaways. This issue adds the detection net so the next novel runaway surface is caught on day one instead of running unseen — #860 ran for four days with no human noticing.

Problem

There is no proactive signal for a PR accumulating abnormal automated activity. Detection is pull (someone happens to look), not push.

Fix

Extend daily-pr-review-health / scripts/fleet_monitor.sh (and/or scripts/pr_review_health.sh) to flag any open PR exceeding soft thresholds as a runaway candidate, with a direct link and the triggering metric:

  • > 50 commits, or
  • > 200 comments, or
  • > N automated cycles (reuse compute_pr_automation_cycles from scripts/lib/pr-automation-budget.sh), or
  • open > 48h with active agent churn.

Thresholds env-overridable. Output via the existing health-report mechanism (issue/label health-check + automated-report, as in #916's report). Detection only — it should not itself mutate PRs.

Acceptance criteria

  • The daily health check lists any open PR over the thresholds, with link + metric.
  • Thresholds are env-overridable with sane defaults.
  • No false positive on a normal, converging PR (validate against current open PRs).
  • Unit coverage for the threshold logic (bats).

References

Activity

  1. github-actions commented on Jun 26, 2026

    @github-actions
    Contributor

    Thanks for the detailed write-up and for connecting this back to the #860 incident — the context around detection vs. prevention is really helpful for scoping the work. Before we dig in: for the automated-cycles threshold specifically, do you have a target value for N in mind (or a formula based on PR age/size), or should the implementer propose a default based on what would have flagged #860 on day one?

  2. don-petry commented on Jul 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #948 — P2 (#860 follow-up): runaway-PR detection in daily health check (commits/comments/cycles/age thresholds)

    Scope

    Add a detection-only runaway-PR net to the daily health check: scan open PRs and flag any exceeding soft thresholds (commits / comments / automated cycles / age-with-churn) as a runaway candidate, surfaced via the existing health-check + automated-report issue mechanism. No PR mutation.

    Design

    • Pure threshold logic in a new sourced lib (scripts/lib/pr-runaway-detect.sh), reusing compute_pr_automation_cycles from scripts/lib/pr-automation-budget.sh for the cycles metric — the unit under test.
    • gh-API gathering + report rendering in scripts/pr_runaway_scan.sh (no LLM; mirrors the pure-fn / gh-jq idiom of fleet_monitor.sh).
    • Thresholds env-overridable with sane defaults: RUNAWAY_MAX_COMMITS=50, RUNAWAY_MAX_COMMENTS=200, RUNAWAY_MAX_CYCLES=10, RUNAWAY_MIN_AGE_HOURS=48, RUNAWAY_CHURN_MIN_CYCLES=1 (age criterion only fires with active agent churn, so a normal/quiet aged PR does not false-positive).

    Implementation Checklist

    • scripts/lib/pr-runaway-detect.sh — thresholds + pr_runaway_reasons + is_pr_runaway + generate_runaway_report
    • scripts/pr_runaway_scan.sh — gather open-PR metrics, run detection, write pr_runaway_report.md, set HAS_RUNAWAY (detection only)
    • .github/workflows/daily-pr-review-health.yml — run scan + create health-check/automated-report issue when candidates found
    • .github/workflows/lint.yml — add new bats file to the bats job list

    Tests to Write

    • tests/test_pr_runaway_detect.bats — each threshold fires at boundary (>50, >200, >N, >48h+churn)
    • env-overridable defaults respected
    • no false positive on a normal converging PR (few commits/comments/cycles, and aged-but-quiet PR)
    • multi-metric PR reports every triggering metric

    Test Command

    scripts/run-bats.sh tests/test_pr_runaway_detect.bats (full suite via lint.yml bats job); lint via bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jul 3, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • scripts/lib/pr-runaway-detect.sh — pure, env-overridable threshold logic (pr_runaway_reasons, is_pr_runaway, pr_age_hours, generate_runaway_report); reuses compute_pr_automation_cycles from the Runaway PR #860: agents self-triggering — add per-PR churn breaker, human-gated cap resets, and self-trigger guards #926 budget lib for the cycles metric
    • scripts/pr_runaway_scan.sh — gh-API + jq scan of every open PR, detection only (no PR mutation); writes pr_runaway_report.md, sets HAS_RUNAWAY/RUNAWAY_COUNT
    • .github/workflows/daily-pr-review-health.yml — runs the scan and opens a health-check+automated-report issue when candidates are found
    • .github/workflows/lint.yml — registered the new bats file
    • tests/test_pr_runaway_detect.bats — 25 unit tests (threshold boundaries, env overrides, false-positive guard, multi-metric, report rendering)

    Acceptance Criteria

    • Daily health check lists any open PR over thresholds, with link + triggering metric (markdown table via generate_runaway_report)
    • Thresholds env-overridable with sane defaults: RUNAWAY_MAX_COMMITS=50, RUNAWAY_MAX_COMMENTS=200, RUNAWAY_MAX_CYCLES=10, RUNAWAY_MIN_AGE_HOURS=48, RUNAWAY_CHURN_MIN_CYCLES=1
    • No false positive on a normal/aged-but-quiet PR — the >48h criterion only fires with active agent churn (unit + end-to-end smoke test confirm PR feat: implement issue #846 — [Phase 2] Cache the LSP index and enforce the 30s cold-start SLA with auto-skip #860 flags, a normal PR does not)
    • Unit coverage for the threshold logic (bats)
    • Detection only — nothing mutates a PR

    Test Results

    ok 18 RUNAWAY_MIN_AGE_HOURS override is respected
    ok 19 RUNAWAY_CHURN_MIN_CYCLES override gates the age criterion
    ok 20 non-numeric inputs degrade to 0 (no crash, no false fire)
    ok 21 missing arguments degrade to 0
    ok 22 pr_age_hours computes whole hours against an injected now
    ok 23 pr_age_hours degrades to 0 on unparseable input
    ok 24 generate_runaway_report renders each candidate with its link and reason
    ok 25 generate_runaway_report on an empty file prints an all-clear line, not a table
    

    Full lint.yml bats list (1018 tests) passes; bash .dev-lead/scripts/dev-lead-lint.sh passes (shellcheck 87 files + agent frontmatter + CODEOWNERS); both modified workflows are valid YAML.

    Files Changed

    • scripts/lib/pr-runaway-detect.sh (new): pure threshold/detection logic + report renderer
    • scripts/pr_runaway_scan.sh (new): open-PR scan orchestration (detection only)
    • tests/test_pr_runaway_detect.bats (new): 25 unit tests
    • .github/workflows/daily-pr-review-health.yml: scan + runaway-issue steps
    • .github/workflows/lint.yml: registered the new test file

    Notes

  4. added a commit that references this issue on Jul 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupenhancementFeature requests

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions