Skip to content

fleet-monitor: fleet-tracker issues never auto-close on recovery/deletion → 107 stale alerts #1203

Closed

Description

@don-petry

Problem

The Actions Fleet Monitor (actions-fleet-monitor.yml, daily) creates/updates a fleet-tracker issue per (repo × workflow) whose failure rate exceeds 10%. Dedup works — it searches by exact title and updates the existing issue in place (stamping Last updated ... on <today>). But it has no close path: when a workflow recovers (drops ≤10%) or is deleted, the monitor simply stops updating the issue and leaves it open forever.

Result: 107 open fleet-tracker issues, most for workflows that recovered long ago. Evidence — several carry Last updated stamps from weeks/months back while the monitor runs daily:

  • google-app-scripts#304 — stamp 2026-05-19 (~55 days stale)
  • broodly#224 — 2026-05-20
  • .github#334 — 2026-05-21

Plus 4 issues for claude.yml / claude-code-reusable.yml — a workflow deleted in #456. The monitor only scans state == "active" workflows, so it can never re-evaluate (or close) an issue for a deleted workflow.

This is the same class as the Scorecard "only closes at 10/10" and ContentTwin dedup bugs.

Fix (implemented — PR linked below)

Add a "Close resolved fleet-tracker issues" step after the tracking step: close any open fleet-tracker issue whose Last updated ... on <date> stamp is older than STALE_DAYS (default 3 → 3 missed daily runs), with a ✅ Auto-resolved comment. Reopens automatically (fresh issue) if the workflow crosses the threshold again.

Safety:

  • Gated on fleet_high_failure.json existing → a failed scan can't mass-close.
  • Staleness uses each issue's own stamp, not the current run, so a single bad-scan day never false-closes (stamp is only 1 day old).
  • Comment + close are each try/catch (a comment-capped issue still closes).

One-time cleanup

After merge, the next daily run auto-closes the whole recovered/deleted backlog. The 4 deleted-claude.yml issues are closed immediately (manually) since a deleted workflow won't be re-scanned.

Acceptance

  • A fleet-tracker issue closes automatically once its workflow has been healthy/absent for >STALE_DAYS.
  • Still-failing workflows keep their issue open (fresh stamp).
  • No mass-close when a scan fails.

Activity

  1. github-actions commented on Jul 14, 2026

    @github-actions
    Contributor

    Thanks for the thorough write-up and for already having a fix ready — the staleness-based auto-close approach is well thought out. Before we review the PR, could you confirm whether the STALE_DAYS default of 3 is meant to be configurable via a workflow input, or is it a hard-coded constant in the script?

  2. don-petry commented on Jul 14, 2026

    @don-petry
    CollaboratorAuthor

    Digest accumulation (health-check-only issues) folded in: fleet-monitor daily digest close-prior added to #1204; pr-review-health + runaway digests fixed in #1212. One-time cleanup of ~66 stale digests in progress.

  3. don-petry commented on Oct 21, 2026

    @don-petry
    CollaboratorAuthor

    Closing after the 2026-10-21 backlog triage.

    Why: Fixed: fleet-tracker auto-close on recovery shipped.
    Evidence: PR #1204 merged 2026-07-14

    Reopen if this still reproduces on main.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions