Repository navigation
Description
Follow-up to #1066, and strictly worse than the dwell reset that issue describes. #1066 says a busy agent's dwell restarts each tick. This is about the frontier itself resetting, which means outer rings can be unreachable indefinitely rather than merely delayed.
The mechanism
cmd_promote advances exactly one ring per invocation: the frontier, defined in scripts/canary-rollout.sh (~L921) as the first channel in ring order whose commit differs from the candidate:
if [ "$ch" = "next" ] || [ "$c" = "$cand" ]; then :; else frontier="$ch"; break; fiThe candidate is whatever <agent>/v<M>-next currently points at. So every autocut that advances next makes ring0 differ again, and the frontier snaps back to ring0 — regardless of how far the outer rings had progressed.
Observed
dev-lead, 2026-09-08. ring0 and ring1 had both been walked up to 90dcc01c, leaving stable as the only lagging ring:
v139-next 27b0e8a3 <- autocut moved it
v139-ring0 90dcc01c <- now differs from candidate => frontier resets HERE
v139-ring1 90dcc01c
v139-stable 40f6c232 <- the ring we actually needed to advance
promote reported gate=SOAKING for ring 'ring0' [next->ring0]. stable was three gated promotions away again, having been one away moments earlier.
Why this can be unbounded, not just slow
The cumulative dwell from next to stable for dev-lead is 4h + 8h + 12h = 24h (next->ring0 4h, ring0->ring1 8h, ring1->stable 12h). The canary schedule is cron: "33 */4 * * *" — every 4 hours. An agent whose watched paths change on most ticks therefore resets the frontier roughly every 4 hours, while needing 24 hours of uninterrupted progress to drain the pipeline.
For an active agent, stable is not merely delayed — it may be unreachable by normal promotion. dev-lead/v139-stable sat on 40f6c232 (2026-08-18) for three weeks while next advanced through eight releases (v139.4.0 → v139.11.0).
Second-order effect: it forces bad choices
Because the pipeline cannot drain, the only way to move stable is promote --override, repeatedly, walking each ring. That has two costs:
- It discards the soak evidence you were waiting for. On 2026-09-08 the fleet had 152 clean runs on
90dcc01cacross three canary repos. Override-walking shipped9e709367— a commit with zero canary coverage — because each promote advances the target ring to the current candidate, not to the commit that was actually soaked. - It normalises
--override, which is meant to be exceptional.
Acceptance criteria
- Advancing
nextmust not reset progress already made on outer rings. Options: pin the frontier to a specific candidate for the duration of its rollout; allow multiple rings to advance per run when each one's own gate passes; or track per-ring target versions rather than deriving everything from the single current candidate. - It must be possible to promote a specific ring to a specific soaked release, without
--overrideand without walking the whole chain. Notecmd_rollbacklooks like this but is unusable for v-scoped channels — see the sibling issue on its major-scope bug. - Cumulative dwell to
stablemust be reachable given the schedule interval, or the mismatch must be surfaced explicitly (e.g. "pipeline cannot drain at current cut cadence") rather than silently never completing. - Regression test on the observed shape: rings walked to a candidate, then
nextadvanced — outer-ring progress must not be discarded.
Related
- Autocut runs before promote-all in the same job, so a busy agent's dwell resets every tick and can never promote — dev-lead stuck at next since 08-12 (blocks #1592) #1066 — dwell reset (the milder sibling; same root cause, autocut and promote sharing one run)
- Gate soak is vacuous: dwell/sample count runs from repos that are not executing the candidate — and dev-lead's 'next' tier is structurally unoccupiable (SC2 forbids its only member) #1086 — soak measured on repos not executing the candidate
petry-projects/.github-private#1592— the delivery gap this whole family blocks
Activity
don-petry commentedon Oct 21, 2026
Closing after the 2026-10-21 backlog triage.
Why: Fixed: every ring pair evaluated independently, frontier no longer resets.
Evidence: PR #1220 (ef7bfdb); cr.sh _frontier_state L2111
Reopen if this still reproduces on main.