Skip to content

[Fleet Monitor] petry-projects/.github β€” .github/workflows/canary-rollout.ymlΒ #939

Closed

Description

@don-petry

πŸ”΄ CRITICAL: petry-projects/.github / .github/workflows/canary-rollout.yml

Metric Value
Failure Rate 57.1% (4 / 7 runs)
Status CRITICAL
p50 Duration 827s
p95 Duration 18885s
Successful Runs 2
Cancelled Runs 1

Workflow: .github/workflows/canary-rollout.yml

Last updated by Fleet Monitor run on 2026-08-07.
Threshold: failure rate > 10% over the monitored window.

Activity

  1. don-petry commented on Aug 5, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #939 β€” [Fleet Monitor] petry-projects/.github β€” .github/workflows/canary-rollout.yml

    Root Cause

    The canary job in .github/workflows/canary-rollout.yml sets timeout-minutes: 15 (900s), but the Fleet Monitor reports p50 = 877s and p95 = 988s. Half the scheduled fleet sweeps already finish within ~23s of the ceiling and the tail (p95) exceeds it, so GitHub kills those runs at the timeout β€” which matches the observed 50% failure rate (3/6) plus the 1 cancelled run. The fleet-wide promote-all/evaluate-all sweep reads run-history owner-wide, so its runtime grows with the fleet; a 15-minute bound is simply too tight.

    Scope

    Raise the canary job's timeout-minutes to give durable headroom over the observed p95, and add a bats regression guard so the timeout can't silently regress back below that headroom.

    Implementation Checklist

    • .github/workflows/canary-rollout.yml β€” raise canary job timeout-minutes from 15 to 30
    • .github/workflows/canary-rollout-tests.yml β€” add .github/workflows/canary-rollout.yml to the trigger paths so the guard runs when the workflow timeout is edited

    Tests to Write

    • tests/canary_rollout.bats β€” regression guard: the canary job declares timeout-minutes and it is >= 20 (documented floor giving headroom over the observed ~16.5-min p95). Fails against the current value of 15.

    Test Command

    bats --print-output-on-failure tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh)

  2. don-petry commented on Aug 5, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Root Cause

    The canary job's timeout-minutes: 15 (900s) was too tight for the scheduled fleet sweep, whose runtime grows with the fleet (owner-wide run-history reads). With observed p50 = 877s and p95 = 988s, the tail of each sweep was being killed at the 900s ceiling β€” matching the 50% failure rate (3/6) plus the 1 cancelled run.

    Plan Execution

    • Added a bats regression guard (red β†’ green against the fix) asserting the canary job's timeout-minutes keeps headroom over the observed p95
    • Raised the canary job timeout-minutes from 15 to 30
    • Added .github/workflows/canary-rollout.yml to the tests workflow trigger paths so the guard runs whenever the timeout is edited
    • Full bats suite + repo lint pass

    Test Results

    251 canary-rollout.yml: canary job declares a timeout-minutes ... ok
    252 canary-rollout.yml: canary job timeout-minutes keeps headroom over the observed p95 (#939) ... ok
    
    252 tests, 0 failures
    
    dev-lead-lint.sh: shellcheck on 29 file(s)... OK; CODEOWNERS OK; all checks passed
    YAML parse: OK (both edited workflows)
    

    Files Changed

    • .github/workflows/canary-rollout.yml: timeout-minutes 15 β†’ 30 on the canary job (with rationale comment citing the observed p50/p95).
    • .github/workflows/canary-rollout-tests.yml: added .github/workflows/canary-rollout.yml to pull_request/push trigger paths so the guard runs when the workflow timeout changes.
    • tests/canary_rollout.bats: added two regression tests guarding the timeout headroom ([Fleet Monitor] petry-projects/.github β€” .github/workflows/canary-rollout.ymlΒ #939).

    Notes

    • The fix addresses the timeout-induced failures, which the metrics point to (p95 988s > 900s). If future failures persist despite the raised bound, they would indicate a separate, non-timeout cause worth a follow-up investigation.
    • The regression guard floor is 20 min (documented headroom over the ~16.5-min p95); the actual value is set to 30 min for room as the fleet grows.
  3. added 2 commits that reference this issue on Aug 7, 2026
  4. added a commit that references this issue on Sep 7, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions