Repository navigation
Description
Summary
canary-rollout.yml -f command=promote has no way to say which commit or which ring the operator approved. It resolves the candidate as "whatever <agent>/next points at" and the ring as "the first ring not on that commit", both when the run executes. A dry run therefore verifies nothing about the real run that follows it: if next is re-cut in between, the real run silently does something different, and with override=true it force-moves a tag regardless of gate state.
The gap between the two is not small. A dispatched run can sit queued for many minutes behind org runner contention, and the scheduled tick re-cuts next every 4 hours whenever a watched path changed on the host's main.
Mechanism
Permalinks at cd0b1675:
scripts/canary-rollout.shL1008-L1011:_frontier_statereadscand="$(channel_commit "$agent" next)"and picks the frontier as the first ring whose commit differs from it.scripts/canary-rollout.shL1246 and L1301-L1304:cmd_promotecalls it fresh, then either prints the[DRY-RUN] would: … sha=$candline or moves the tag to that same freshly-read$cand..github/workflows/canary-rollout.ymlL64-L118: the dispatch inputs arecommand,agent,override,confirm,dry_run, plusringandto, which onlyrollbackuses.rollbackis pinned to an immutable release;promoteis pinned to nothing.- The manual promote runs in concurrency group
canary-rollout-<agent>and the scheduled run incanary-rollout-fleet, so the scheduled autocut and a manual promote for the same agent do not serialize.
So after a re-cut, every ring is "not on the candidate" again and the frontier snaps back to ring0. A run the operator dispatched to move stable to commit A instead moves ring0 to commit B, exits green, and logs promoted.
Evidence (dev-lead, 2026-10-01, one-off override release to v139.39.0 = 51cb28ee)
All times UTC.
| Time | Event |
|---|---|
| 20:01:43 | combsmith/.github-private#1977 merges to main (fb99072c). It changes scripts/template_stub_drift.sh; scripts/ is a default watched agent_ref_paths entry, so the next scheduled autocut will re-cut dev-lead/v139-next. |
| 20:04:22 | Dry run 36918476004 prints [DRY-RUN] would: gh api PATCH …/tags/dev-lead/v139-stable sha=51cb28ee…. |
| 20:04:35 | Real run 36918930731 dispatched (override=true, dry_run=false). It stays queued: the #1977 merge set off a rebase storm, with 326 runs queued in .github-private at 20:11 and 78-86 in .github. |
| 20:25:05 | The run finally executes and logs promoted dev-lead/v139-stable -> 51cb28ee6c36. |
| 20:33 | Next 33 */4 * * * scheduled tick is due. |
The run was queued for about 20 minutes and executed about 8 minutes before the tick that would re-cut next. Had the order been reversed, the run would have force-moved dev-lead/v139-ring0 to fb99072c, a commit that was never dry-run, never soaked and never looked at, while stable stayed where it was and the run reported success.
It came out right only because the operator noticed the merge, polled the next tag every 15 seconds and was ready to cancel the queued run. Nothing in the tooling provides that guard.
The same exposure exists without any queueing: there is always a gap between a dry run and the real dispatch, and the 2026-10-01 release needed three dry-run/real pairs (one per ring).
Acceptance criteria
promoteaccepts an expected candidate commit and an expected ring, as a CLI flag and asworkflow_dispatchinputs (for exampleexpect_shaandexpect_ring). A prefix of at least 8 hex characters is accepted for the commit.- When either is supplied and the live candidate or the computed frontier differs,
promotemoves nothing, prints an::error::naming the expected and actual values, and exits non-zero so the run is red. override=truewithdry_run=falseand no expected commit is refused with a clear error. An override bypasses the gate, so it must at least name what it is overriding for.- The dry-run output prints the exact follow-up dispatch command, including the expected commit and ring it just computed, so the real run can be copy-pasted.
promote-all(the scheduled arm) is unchanged: it runs autocut and promotion in one job by design.tests/canary_rollout.batscovers: candidate changed between dry run and real run (refuses, no tag write), frontier changed (refuses, no tag write), both match (promotes), and override without an expected commit (refuses).- The canary runbook's manual-promotion section documents the dry-run then pinned-real-run sequence.
Related, not duplicates
- .github#1118 "canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation": the same "newest
nextwins" model, seen from the gate's side. Fixing it may change how the candidate is chosen, but an operator still needs to pin what they approved. - .github#1175 "Canary Rollout times out: 6 of the last 7 scheduled runs hit timeout-minutes: 30": the same runner contention that widened the window here.
- .github#1177 "Canary Rollout operator gaps: no allow_pre_existing dispatch input, 'all agents promoted' summary despite BLOCKED gates": other missing dispatch inputs and misleading output on the same workflow.
Activity
don-petry commentedon Oct 10, 2026
Second occurrence, 2026-10-10 (dev-lead). A maintainer-approved one-time promotion of next to ring0 was dispatched for release v139.51.1 (commit d7470703), verified on next at 02:25 UTC. The scheduled autocut then re-cut next at 02:37:33 UTC as v139.52.0 (commit 71920a24, four commits further on), and the queued promote ran afterwards and moved ring0 to the NEW next, not the commit that had been verified and approved. ring1 and stable were unchanged. The four extra commits had not soaked on next at all (it had existed for minutes). This is the behaviour this issue describes: promote targets whatever next is when the run executes. A promote that is approved against a specific commit therefore cannot be guaranteed to move that commit. Suggested direction, consistent with this issue: let the dispatch take an expected-commit input and refuse to move if next differs, which would also have turned this case into a clean stop.