Skip to content

[Fleet Monitor] petry-projects/.github-private β€” .github/workflows/test-deletion-guard.ymlΒ #941

Closed

Description

@don-petry

🟑 WARNING: petry-projects/.github-private / .github/workflows/test-deletion-guard.yml

Metric Value
Failure Rate 11.1% (2 / 18 runs)
Status WARNING
p50 Duration 11s
p95 Duration 82s
Successful Runs 16
Cancelled Runs 0

Workflow: .github/workflows/test-deletion-guard.yml

Last updated by Fleet Monitor run on 2026-06-26.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. added
    health-checkAutomated health check report
    dev-leadFor dev-lead agent pickup
    fleet-trackerTracked by the Actions Fleet Monitor
    on Jun 26, 2026
  2. don-petry commented on Jun 26, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead Implementation Plan

    Issue: #941 β€” [Fleet Monitor] petry-projects/.github-private β€” .github/workflows/test-deletion-guard.yml

    Root Cause

    The two "failures" flagged by Fleet Monitor are the guard working as designed: a PR on claude/auto-implementation-blockers-oukz80 deleted files under tests/ without the ack-test-deletion label, so the guard correctly exited 1 to block the merge (runs 28166896375, 28166721237). These are intentional policy-gate failures, not flakiness or breakage. The guard itself must not be weakened (AGENTS.md: "Do not remove either guard").

    The real defect is in the Fleet Monitor: its high-failure tracker (scripts/fleet_monitor.sh Β§3b) counts any failure conclusion toward the >10% rate, so a gate workflow whose job is to fail bad PRs generates false-positive tracker issues like this one.

    Scope

    Exclude documented gate workflows (whose failure is intentional enforcement) from the Fleet Monitor's high-failure issue tracking, while keeping them visible in the report and still surfacing genuine monitor ERROR rows.

    Implementation Checklist

    • scripts/fleet_report.sh β€” add FLEET_GATE_WORKFLOWS constant + extract the Β§3b inline jq into a pure, testable filter_high_failure <metrics_file> function that excludes gate workflows (matched by basename) from rate-based tracking (ERROR rows still surface)
    • scripts/fleet_monitor.sh β€” replace the inline Β§3b jq with a call to filter_high_failure
    • AGENTS.md β€” note that gate workflows are exempt from Fleet Monitor high-failure tracking via FLEET_GATE_WORKFLOWS

    Tests to Write

    • tests/fleet_report.bats β€” gate workflow with >10% failure rate is excluded from filter_high_failure output
    • non-gate workflow with >10% failure rate is still included
    • gate matching works on full workflow path (.github/workflows/test-deletion-guard.yml) and bare basename
    • an ERROR row for a gate workflow is still surfaced (monitor read failure β‰  intentional gate failure)
    • existing keep/drop logic (LOW-CONF downgrade, >10% threshold) preserved

    Test Command

    scripts/run-bats.sh tests/fleet_report.bats (or npx bats tests/fleet_report.bats), then bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jun 26, 2026

    @don-petry
    CollaboratorAuthor

    Dev-Lead: Implementation Complete

    Root Cause (confirmed)

    The two flagged "failures" of test-deletion-guard.yml (runs 28166896375, 28166721237, both on claude/auto-implementation-blockers-oukz80) were the guard working as designed β€” the PR deleted tests/dev-lead/unit/test_lsp_coldstart_sla.bats, tests/test_pr_automation_budget.bats, and tests/test_pr_runaway_regression.bats without the ack-test-deletion label, so the guard correctly exited 1 to block the merge. These are intentional policy-gate failures, not flakiness or breakage; the guard must not be weakened (AGENTS.md).

    The actual defect is in the Fleet Monitor: its high-failure tracker (scripts/fleet_monitor.sh Β§3b) counted any failure conclusion toward the >10% rate, so a gate workflow whose job is to fail bad PRs produced a false-positive tracker issue (this one).

    Plan Execution

    • Added FLEET_GATE_WORKFLOWS constant in scripts/fleet_report.sh (test-deletion-guard.yml holdout-guard.yml)
    • Extracted the Β§3b inline jq into a pure, testable filter_high_failure <metrics_file> function that excludes gate workflows (matched by basename) from rate-based tracking, while still surfacing genuine monitor ERROR rows
    • Wired scripts/fleet_monitor.sh Β§3b to call filter_high_failure
    • Documented the exemption in AGENTS.md (test-deletion-guard.yml exception block)
    • Added 6 bats tests covering exclusion, basename matching, non-gate tracking, ERROR surfacing, LOW-CONF drop, and a mixed-rows case

    Test Results

    $ npx bats tests/fleet_report.bats
    1..46
    ok 41 filter_high_failure: gate workflow over 10% is excluded from tracking
    ok 42 filter_high_failure: gate matched by bare basename is excluded
    ok 43 filter_high_failure: non-gate workflow over 10% is still tracked
    ok 44 filter_high_failure: ERROR row for a gate workflow is still surfaced
    ok 45 filter_high_failure: CRITICAL with under 5 runs is dropped as LOW-CONF
    ok 46 filter_high_failure: mixed rows keep only the tracked non-gate workflow
    (all 46 pass)
    
    $ npx bats tests/fleet_stub_drift.bats   β†’ 1..16 (all pass)
    
    $ bash .dev-lead/scripts/dev-lead-lint.sh
      [lint] shellcheck on 79 file(s)...
      [lint] validating 4 agent profile(s)...
      [lint] all checks passed
    
    $ shellcheck -S warning scripts/fleet_report.sh scripts/fleet_monitor.sh  β†’ clean
    

    Files Changed

    • scripts/fleet_report.sh: new FLEET_GATE_WORKFLOWS constant + pure filter_high_failure function with gate exclusion
    • scripts/fleet_monitor.sh: Β§3b now calls filter_high_failure (removes the duplicated inline jq)
    • tests/fleet_report.bats: 6 new tests for the function
    • AGENTS.md: documents the gate exemption from Fleet Monitor high-failure tracking

    Notes

    • Gate workflows still appear in the human-readable Fleet report (scorecard / Fleet Detail) for visibility β€” only the issue-tracking JSON export excludes them, so no monitoring blind spot is created.
    • A gate workflow's ERROR row (monitor could not read its runs) is still surfaced β€” a read failure is a monitor problem, not the gate doing its job.
    • holdout-guard.yml was included alongside test-deletion-guard.yml because it is the identical gate class (it fails the proposer's PR by design) and would otherwise produce the same false-positive next time it blocks a PR.
    • No change to engine/model routing or to the guard workflow itself.
  4. added a commit that references this issue on Jun 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickupfleet-trackerTracked by the Actions Fleet Monitorhealth-checkAutomated health check report

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions