You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The canary job in .github/workflows/canary-rollout.yml sets timeout-minutes: 15 (900s), but the Fleet Monitor reports p50 = 877s and p95 = 988s. Half the scheduled fleet sweeps already finish within ~23s of the ceiling and the tail (p95) exceeds it, so GitHub kills those runs at the timeout β which matches the observed 50% failure rate (3/6) plus the 1 cancelled run. The fleet-wide promote-all/evaluate-all sweep reads run-history owner-wide, so its runtime grows with the fleet; a 15-minute bound is simply too tight.
Scope
Raise the canary job's timeout-minutes to give durable headroom over the observed p95, and add a bats regression guard so the timeout can't silently regress back below that headroom.
Implementation Checklist
.github/workflows/canary-rollout.yml β raise canary job timeout-minutes from 15 to 30
.github/workflows/canary-rollout-tests.yml β add .github/workflows/canary-rollout.yml to the trigger paths so the guard runs when the workflow timeout is edited
Tests to Write
tests/canary_rollout.bats β regression guard: the canary job declares timeout-minutes and it is >= 20 (documented floor giving headroom over the observed ~16.5-min p95). Fails against the current value of 15.
The canary job's timeout-minutes: 15 (900s) was too tight for the scheduled fleet sweep, whose runtime grows with the fleet (owner-wide run-history reads). With observed p50 = 877s and p95 = 988s, the tail of each sweep was being killed at the 900s ceiling β matching the 50% failure rate (3/6) plus the 1 cancelled run.
Plan Execution
Added a bats regression guard (red β green against the fix) asserting the canary job's timeout-minutes keeps headroom over the observed p95
Raised the canary job timeout-minutes from 15 to 30
Added .github/workflows/canary-rollout.yml to the tests workflow trigger paths so the guard runs whenever the timeout is edited
Full bats suite + repo lint pass
Test Results
251 canary-rollout.yml: canary job declares a timeout-minutes ... ok
252 canary-rollout.yml: canary job timeout-minutes keeps headroom over the observed p95 (#939) ... ok
252 tests, 0 failures
dev-lead-lint.sh: shellcheck on 29 file(s)... OK; CODEOWNERS OK; all checks passed
YAML parse: OK (both edited workflows)
Files Changed
.github/workflows/canary-rollout.yml: timeout-minutes 15 β 30 on the canary job (with rationale comment citing the observed p50/p95).
.github/workflows/canary-rollout-tests.yml: added .github/workflows/canary-rollout.yml to pull_request/push trigger paths so the guard runs when the workflow timeout changes.
The fix addresses the timeout-induced failures, which the metrics point to (p95 988s > 900s). If future failures persist despite the raised bound, they would indicate a separate, non-timeout cause worth a follow-up investigation.
The regression guard floor is 20 min (documented headroom over the ~16.5-min p95); the actual value is set to 30 min for room as the fleet grows.
Activity
don-petry commentedon Aug 5, 2026
Dev-Lead Implementation Plan
Issue: #939 β [Fleet Monitor] petry-projects/.github β .github/workflows/canary-rollout.yml
Root Cause
The
canaryjob in.github/workflows/canary-rollout.ymlsetstimeout-minutes: 15(900s), but the Fleet Monitor reports p50 = 877s and p95 = 988s. Half the scheduled fleet sweeps already finish within ~23s of the ceiling and the tail (p95) exceeds it, so GitHub kills those runs at the timeout β which matches the observed 50% failure rate (3/6) plus the 1 cancelled run. The fleet-widepromote-all/evaluate-allsweep reads run-history owner-wide, so its runtime grows with the fleet; a 15-minute bound is simply too tight.Scope
Raise the
canaryjob'stimeout-minutesto give durable headroom over the observed p95, and add a bats regression guard so the timeout can't silently regress back below that headroom.Implementation Checklist
.github/workflows/canary-rollout.ymlβ raisecanaryjobtimeout-minutesfrom15to30.github/workflows/canary-rollout-tests.ymlβ add.github/workflows/canary-rollout.ymlto the triggerpathsso the guard runs when the workflow timeout is editedTests to Write
tests/canary_rollout.batsβ regression guard: thecanaryjob declarestimeout-minutesand it is >= 20 (documented floor giving headroom over the observed ~16.5-min p95). Fails against the current value of 15.Test Command
bats --print-output-on-failure tests/canary_rollout.bats(plusbash .dev-lead/scripts/dev-lead-lint.sh)don-petry commentedon Aug 5, 2026
Dev-Lead: Implementation Complete
Root Cause
The
canaryjob'stimeout-minutes: 15(900s) was too tight for the scheduled fleet sweep, whose runtime grows with the fleet (owner-wide run-history reads). With observed p50 = 877s and p95 = 988s, the tail of each sweep was being killed at the 900s ceiling β matching the 50% failure rate (3/6) plus the 1 cancelled run.Plan Execution
timeout-minuteskeeps headroom over the observed p95canaryjobtimeout-minutesfrom15to30.github/workflows/canary-rollout.ymlto the tests workflow triggerpathsso the guard runs whenever the timeout is editedTest Results
Files Changed
.github/workflows/canary-rollout.yml:timeout-minutes15 β 30 on thecanaryjob (with rationale comment citing the observed p50/p95)..github/workflows/canary-rollout-tests.yml: added.github/workflows/canary-rollout.ymltopull_request/pushtrigger paths so the guard runs when the workflow timeout changes.tests/canary_rollout.bats: added two regression tests guarding the timeout headroom ([Fleet Monitor] petry-projects/.github β .github/workflows/canary-rollout.ymlΒ #939).Notes
feat: implement issue #939 β [Fleet Monitor] petry-projects/.github ββ¦
feat: implement issue #939 β [Fleet Monitor] petry-projects/.github ββ¦
feat: implement issue #939 β [Fleet Monitor] petry-projects/.github ββ¦