Skip to content

canary-rollout: with two candidates in flight, no command can advance an upper ring pair past a BLOCKED gate, and an API-outage failure is a permanent REGRESSION #1324

Open

Description

@don-petry

Summary

Once autocut re-cuts next, an operator has no working command to advance an upper ring pair
past a BLOCKED gate. Two things combine:

  1. promote --override applies to the lowest pending pair only.
  2. A failure caused by an outage (here: the shared GitHub API quota running out) is triaged
    REGRESSION and stays counted for the whole life of that candidate, so waiting never clears it.

The candidate is then stuck below stable until a different commit reaches the ring beneath it.

Observed, 2026-10-11 to 2026-10-12 (dev-lead)

dev-lead/v139-next    5797da5f  (v139.63.1, cut ~23:54Z by the scheduled autocut)
dev-lead/v139-ring0   684e4b9d  (v139.62.0, cut 17:15Z)
dev-lead/v139-ring1   684e4b9d
dev-lead/v139-stable  79fe57ba  (v139.59.0)

Scheduled sweep, run 38186007463 (23:54Z):

gate=SOAKING for ring 'ring0' [next->ring0] (cum_fail=0, triage=-) — not promoting.
##[error]gate=BLOCKED (triage=REGRESSION) for 'stable' [ring1->stable] — candidate regression suspected; not promoting.

The counted failures are listed in #664 "Canary blocker: dev-lead ring1->stable (cum_fail=1, REGRESSION)":

Run Release it ran Failure
petry-projects/.github-private run 38164859420 (18:41Z) 684e4b9d fetch_pr_context: failed to fetch CI check-runs, preceded by budget event collection incomplete … API unavailable and two could not list … comments warnings
petry-projects/.github-private run 38164745202 (18:39–18:41Z) 684e4b9d exit 1 directly after stamp_pass_replies: could not read the pending-replies marker; the exact failing call was not identified
petry-projects/TalkTerm run 38161694364 (18:02Z) dd5f0dca (previous release) API rate limit exceeded for user ID …, HTTP 403

Both candidate-SHA failures are on the same PR, in the same minute that the shared quota ran out
(combsmith/.github-private#2197 "The fleet and every session share one GitHub API quota").

Gap 1 — --override cannot reach an upper pair

cmd_promote evaluates every ring pair independently (#1118) but grants the override to the first
pending pair only:

https://github.com/petry-projects/.github/blob/452ea69655bb1cbd1283f95be21210ec614b2f43/scripts/canary-rollout.sh#L2304-L2309

That is right for a single in-flight candidate. With two candidates in flight, the first pending pair
is next->ring0, so promote dev-lead --override would move the unsoaked 5797da5f to ring0 and
leave ring1->stable blocked. --confirm and --allow-pre-existing do not advance a REGRESSION
either, by design.

The remaining lever, rollback <agent> stable --to <release>, would pin the exact commit, but it
moves the bare <agent>/stable tag rather than <agent>/v<M>-stable (#1103 "cmd_rollback moves the
bare / tag"), so it changes nothing consumers read.

This is the operator-side remainder of #1102 "Frontier resets to ring0 whenever autocut advances
'next'": the gate now keeps evaluating the outer pair, but a human still cannot act on it.

Gap 2 — an outage failure is a permanent REGRESSION for that candidate

cum_fail is cumulative over the per-candidate window, and a failed run that executed the candidate
with a changed reusable is triaged REGRESSION whatever the cause. A GitHub REST 403 or an
unreachable API in fetch_pr_context is not something the candidate can cause. Same family as #1057
"canary gate counts engine-quota failures … as cum_fail", which covers the Claude 429 case; this is
the GitHub API quota case.

Acceptance criteria

  • An operator can direct an override at a named transition (for example transition=ring1->stable),
    and the dry run prints the exact tag and commit it would move. Without a named transition the
    behaviour is unchanged.
  • An override of an upper pair never advances any other pending pair in the same run.
  • A failed run whose failing step shows a GitHub API rate-limit or API-unavailable signature is not
    triaged REGRESSION; it is reported as its own class in the blocker issue so a human can see it.
  • Tests cover two candidates in flight with the upper pair blocked, for both the default and the
    named-transition override.

Not in scope

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugBug reports

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions