Repository navigation
Description
This is why dev-lead's channel tags have not advanced since 2026-08-12, and it blocks petry-projects/.github-private#1592. The gate is healthy and promote-all works — the scheduled job simply cannot ever promote a busy agent.
The mechanism
.github/workflows/canary-rollout.yml runs both stages in one job, in this order:
L219 - name: Autocut new candidates (front end) <- cuts a new release, moves <agent>/v<M>-next
L231 - name: Run canary-rollout <- CMD=promote-all on the schedule
promote-all measures dwell from the candidate's own cut date (candidate_cut_date, the per-candidate window from #548). So when autocut cuts on a tick, promote-all — running seconds later in the same job — evaluates a candidate that is seconds old against dwell_hours: 4. It reports SOAKING and declines to promote, every time.
Directly observed, run 33632696092:
canary run created 2026-09-02T12:55:41Z
dev-lead/v139.7.0 cut 2026-09-02T12:56:14Z <- cut by THAT RUN's autocut step
promote-all verdict: gate=SOAKING for ring 'ring0' [next->ring0] (cum_fail=0, triage=-)
The candidate was 33 seconds old when its 4-hour dwell was assessed.
Why it never recovers on a later tick
The schedule is cron: "33 */4 * * *" — every 4 hours — and dwell_hours is 4. So the dwell requirement and the tick interval are equal. For an agent whose watched paths change at least once per 4-hour window, autocut cuts on essentially every tick, and each cut resets the clock before promote-all can ever observe a 4-hour-old candidate.
dev-lead is the busiest agent in the fleet, so it cuts nearly every tick. Recent cuts:
v139.4.0 2026-08-31T23:37:09Z
v139.5.0 2026-09-01T09:34:34Z
v139.6.0 2026-09-01T16:51:49Z
v139.7.0 2026-09-02T12:56:14Z
A manual evaluate at 2026-09-02T10:59Z showed dwell=1h/4h against the v139.5.0 candidate — it was accumulating correctly and would have qualified at ~13:34Z. The 12:56 autocut reset it to zero first.
This is self-reinforcing in the wrong direction: the more active an agent is, the less able it is to promote. Quiet agents promote fine — the same run advanced auto-rebase/v2-ring0.
Not to be confused with
- Not a gate failure.
cum_fail=0,triage=-. The reliability signal is clean. - Not the
#1046cut-date bug (fixed) or a falseREGRESSION(also clear). - Not
--overridematerial. Overriding would bypass the soak entirely, which is the one real safety check. The soak should complete, not be skipped.
Secondary finding: the job also exceeds its own timeout
The same four scheduled runs all end cancelled after ~32 minutes against timeout-minutes: 30 (L136). Step durations from run 33632696092:
Run canary-rollout (promote-all) 997s
Sync blocker issues + fleet-status 887s
Reusable-registry drift audit 18s
Autocut new candidates 40s
≈ 1942s + overhead > 1800s budget
The work completes — steps 7/9/10/11 all report success — so this is not currently losing promotions. But the headroom that #939's test asserts ("timeout-minutes keeps headroom over the observed p95") is gone, every scheduled run is marked cancelled, and a slightly slower tick will start truncating real work.
Acceptance criteria
- A freshly-cut candidate is not evaluated for promotion in the same run that created it. Options, in rough order of preference:
- (a) Split autocut and promote-all into separate scheduled runs, offset so promote-all always sees candidates from a prior tick.
- (b) Have promote-all skip agents whose candidate was cut during this run.
- (c) Decouple the two cadences so the dwell window can close between them.
-
dwell_hoursand the schedule interval must not be equal. With both at 4h, dwell can only ever be satisfied by a tick on which autocut cut nothing — which for an active agent is never. Either shorten the interval, lengthen the dwell, or (preferably) fix the ordering so the two are independent. - Verify against
dev-leadspecifically: after the change,next->ring0must actually reachdwell >= 4hand promote, rather than resetting each tick. - Bring the job back inside
timeout-minutes(or raise it deliberately, with the#939p95 headroom test updated to match). Investigate whySync blocker issues + fleet-status summarytakes ~15 minutes. - Regression test: a candidate cut at time T is not treated as dwell-eligible by a promote-all evaluation at time T + seconds.
Impact
Every agent whose watched paths change at least once per 4-hour window is permanently stuck at next. That is precisely the set of agents that matter most, and it is the delivery half of #1592: fixes merge to main, autocut cuts them onto next, and they never reach the ring the fleet's stubs pin.
Refs petry-projects/.github-private#1592, #548, #939, #1045, #657.
UPDATE 2026-09-07 — softening an overstatement, but the defect stands
I wrote that a busy agent "can never promote". Five days of evidence shows that is too strong. dev-lead has since promoted twice without any code change — canary-rollout.yml is byte-identical to when this was filed (autocut still at L219, promote-all at L231, timeout-minutes: 30 unchanged):
dev-lead/v139-ring0 -> 90dcc01ca854 (2026-09-03)
dev-lead/v139-ring1 -> a7c0f4669e83 (2026-09-02)
So the accurate claim is the one this issue's own analysis implies: promotion is only possible on a tick where autocut cut nothing. Over five days, enough quiet ticks occurred. It is not a permanent block — it is an unpredictable, activity-dependent delay, and the busier the agent the longer it waits. dev-lead sat at next from 2026-08-12 to 2026-09-02 — three weeks — before a quiet tick let it through.
That is still worth fixing, for the reason the delay is invisible: nothing reports "this candidate would have promoted but its dwell was reset". It just silently doesn't happen, which is exactly how #1592 went unnoticed for three weeks.
Also still true and unfixed:
- The scheduled runs were ending
cancelledat ~32 min againsttimeout-minutes: 30. Recent runs (09-05 onward) are completingsuccess, so the pressure has eased, but the#939p95 headroom assertion is worth re-checking rather than assuming it recovered.
Priority accordingly: lower than "promotion is broken", higher than cosmetic. The concrete cost is multi-week, silent delivery lag on the fleet's most active agents.
Activity