Skip to content

Autocut runs before promote-all in the same job, so a busy agent's dwell resets every tick and can never promote — dev-lead stuck at next since 08-12 (blocks #1592) #1066

Open

Description

@don-petry

This is why dev-lead's channel tags have not advanced since 2026-08-12, and it blocks petry-projects/.github-private#1592. The gate is healthy and promote-all works — the scheduled job simply cannot ever promote a busy agent.

The mechanism

.github/workflows/canary-rollout.yml runs both stages in one job, in this order:

L219  - name: Autocut new candidates (front end)     <- cuts a new release, moves <agent>/v<M>-next
L231  - name: Run canary-rollout                     <- CMD=promote-all on the schedule

promote-all measures dwell from the candidate's own cut date (candidate_cut_date, the per-candidate window from #548). So when autocut cuts on a tick, promote-all — running seconds later in the same job — evaluates a candidate that is seconds old against dwell_hours: 4. It reports SOAKING and declines to promote, every time.

Directly observed, run 33632696092:

canary run created   2026-09-02T12:55:41Z
dev-lead/v139.7.0 cut 2026-09-02T12:56:14Z     <- cut by THAT RUN's autocut step
promote-all verdict:  gate=SOAKING for ring 'ring0' [next->ring0] (cum_fail=0, triage=-)

The candidate was 33 seconds old when its 4-hour dwell was assessed.

Why it never recovers on a later tick

The schedule is cron: "33 */4 * * *" — every 4 hours — and dwell_hours is 4. So the dwell requirement and the tick interval are equal. For an agent whose watched paths change at least once per 4-hour window, autocut cuts on essentially every tick, and each cut resets the clock before promote-all can ever observe a 4-hour-old candidate.

dev-lead is the busiest agent in the fleet, so it cuts nearly every tick. Recent cuts:

v139.4.0  2026-08-31T23:37:09Z
v139.5.0  2026-09-01T09:34:34Z
v139.6.0  2026-09-01T16:51:49Z
v139.7.0  2026-09-02T12:56:14Z

A manual evaluate at 2026-09-02T10:59Z showed dwell=1h/4h against the v139.5.0 candidate — it was accumulating correctly and would have qualified at ~13:34Z. The 12:56 autocut reset it to zero first.

This is self-reinforcing in the wrong direction: the more active an agent is, the less able it is to promote. Quiet agents promote fine — the same run advanced auto-rebase/v2-ring0.

Not to be confused with

  • Not a gate failure. cum_fail=0, triage=-. The reliability signal is clean.
  • Not the #1046 cut-date bug (fixed) or a false REGRESSION (also clear).
  • Not --override material. Overriding would bypass the soak entirely, which is the one real safety check. The soak should complete, not be skipped.

Secondary finding: the job also exceeds its own timeout

The same four scheduled runs all end cancelled after ~32 minutes against timeout-minutes: 30 (L136). Step durations from run 33632696092:

Run canary-rollout (promote-all)          997s
Sync blocker issues + fleet-status        887s
Reusable-registry drift audit              18s
Autocut new candidates                     40s
                                    ≈ 1942s + overhead > 1800s budget

The work completes — steps 7/9/10/11 all report success — so this is not currently losing promotions. But the headroom that #939's test asserts ("timeout-minutes keeps headroom over the observed p95") is gone, every scheduled run is marked cancelled, and a slightly slower tick will start truncating real work.

Acceptance criteria

  • A freshly-cut candidate is not evaluated for promotion in the same run that created it. Options, in rough order of preference:
    • (a) Split autocut and promote-all into separate scheduled runs, offset so promote-all always sees candidates from a prior tick.
    • (b) Have promote-all skip agents whose candidate was cut during this run.
    • (c) Decouple the two cadences so the dwell window can close between them.
  • dwell_hours and the schedule interval must not be equal. With both at 4h, dwell can only ever be satisfied by a tick on which autocut cut nothing — which for an active agent is never. Either shorten the interval, lengthen the dwell, or (preferably) fix the ordering so the two are independent.
  • Verify against dev-lead specifically: after the change, next->ring0 must actually reach dwell >= 4h and promote, rather than resetting each tick.
  • Bring the job back inside timeout-minutes (or raise it deliberately, with the #939 p95 headroom test updated to match). Investigate why Sync blocker issues + fleet-status summary takes ~15 minutes.
  • Regression test: a candidate cut at time T is not treated as dwell-eligible by a promote-all evaluation at time T + seconds.

Impact

Every agent whose watched paths change at least once per 4-hour window is permanently stuck at next. That is precisely the set of agents that matter most, and it is the delivery half of #1592: fixes merge to main, autocut cuts them onto next, and they never reach the ring the fleet's stubs pin.

Refs petry-projects/.github-private#1592, #548, #939, #1045, #657.


UPDATE 2026-09-07 — softening an overstatement, but the defect stands

I wrote that a busy agent "can never promote". Five days of evidence shows that is too strong. dev-lead has since promoted twice without any code change — canary-rollout.yml is byte-identical to when this was filed (autocut still at L219, promote-all at L231, timeout-minutes: 30 unchanged):

dev-lead/v139-ring0 -> 90dcc01ca854  (2026-09-03)
dev-lead/v139-ring1 -> a7c0f4669e83  (2026-09-02)

So the accurate claim is the one this issue's own analysis implies: promotion is only possible on a tick where autocut cut nothing. Over five days, enough quiet ticks occurred. It is not a permanent block — it is an unpredictable, activity-dependent delay, and the busier the agent the longer it waits. dev-lead sat at next from 2026-08-12 to 2026-09-02 — three weeks — before a quiet tick let it through.

That is still worth fixing, for the reason the delay is invisible: nothing reports "this candidate would have promoted but its dwell was reset". It just silently doesn't happen, which is exactly how #1592 went unnoticed for three weeks.

Also still true and unfixed:

  • The scheduled runs were ending cancelled at ~32 min against timeout-minutes: 30. Recent runs (09-05 onward) are completing success, so the pressure has eased, but the #939 p95 headroom assertion is worth re-checking rather than assuming it recovered.

Priority accordingly: lower than "promotion is broken", higher than cosmetic. The concrete cost is multi-week, silent delivery lag on the fleet's most active agents.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions