Repository navigation
Description
Summary
Once autocut re-cuts next, an operator has no working command to advance an upper ring pair
past a BLOCKED gate. Two things combine:
promote --overrideapplies to the lowest pending pair only.- A failure caused by an outage (here: the shared GitHub API quota running out) is triaged
REGRESSIONand stays counted for the whole life of that candidate, so waiting never clears it.
The candidate is then stuck below stable until a different commit reaches the ring beneath it.
Observed, 2026-10-11 to 2026-10-12 (dev-lead)
dev-lead/v139-next 5797da5f (v139.63.1, cut ~23:54Z by the scheduled autocut)
dev-lead/v139-ring0 684e4b9d (v139.62.0, cut 17:15Z)
dev-lead/v139-ring1 684e4b9d
dev-lead/v139-stable 79fe57ba (v139.59.0)
Scheduled sweep, run 38186007463 (23:54Z):
gate=SOAKING for ring 'ring0' [next->ring0] (cum_fail=0, triage=-) — not promoting.
##[error]gate=BLOCKED (triage=REGRESSION) for 'stable' [ring1->stable] — candidate regression suspected; not promoting.
The counted failures are listed in #664 "Canary blocker: dev-lead ring1->stable (cum_fail=1, REGRESSION)":
| Run | Release it ran | Failure |
|---|---|---|
| petry-projects/.github-private run 38164859420 (18:41Z) | 684e4b9d |
fetch_pr_context: failed to fetch CI check-runs, preceded by budget event collection incomplete … API unavailable and two could not list … comments warnings |
| petry-projects/.github-private run 38164745202 (18:39–18:41Z) | 684e4b9d |
exit 1 directly after stamp_pass_replies: could not read the pending-replies marker; the exact failing call was not identified |
| petry-projects/TalkTerm run 38161694364 (18:02Z) | dd5f0dca (previous release) |
API rate limit exceeded for user ID …, HTTP 403 |
Both candidate-SHA failures are on the same PR, in the same minute that the shared quota ran out
(combsmith/.github-private#2197 "The fleet and every session share one GitHub API quota").
Gap 1 — --override cannot reach an upper pair
cmd_promote evaluates every ring pair independently (#1118) but grants the override to the first
pending pair only:
That is right for a single in-flight candidate. With two candidates in flight, the first pending pair
is next->ring0, so promote dev-lead --override would move the unsoaked 5797da5f to ring0 and
leave ring1->stable blocked. --confirm and --allow-pre-existing do not advance a REGRESSION
either, by design.
The remaining lever, rollback <agent> stable --to <release>, would pin the exact commit, but it
moves the bare <agent>/stable tag rather than <agent>/v<M>-stable (#1103 "cmd_rollback moves the
bare / tag"), so it changes nothing consumers read.
This is the operator-side remainder of #1102 "Frontier resets to ring0 whenever autocut advances
'next'": the gate now keeps evaluating the outer pair, but a human still cannot act on it.
Gap 2 — an outage failure is a permanent REGRESSION for that candidate
cum_fail is cumulative over the per-candidate window, and a failed run that executed the candidate
with a changed reusable is triaged REGRESSION whatever the cause. A GitHub REST 403 or an
unreachable API in fetch_pr_context is not something the candidate can cause. Same family as #1057
"canary gate counts engine-quota failures … as cum_fail", which covers the Claude 429 case; this is
the GitHub API quota case.
Acceptance criteria
- An operator can direct an override at a named transition (for example
transition=ring1->stable),
and the dry run prints the exact tag and commit it would move. Without a named transition the
behaviour is unchanged. - An override of an upper pair never advances any other pending pair in the same run.
- A failed run whose failing step shows a GitHub API rate-limit or API-unavailable signature is not
triagedREGRESSION; it is reported as its own class in the blocker issue so a human can see it. - Tests cover two candidates in flight with the upper pair blocked, for both the default and the
named-transition override.
Not in scope
- Fixing
cmd_rollback's tag scope (cmd_rollback moves the bare <agent>/<ring> tag, not <agent>/v<M>-<ring> — the emergency lever is a silent no-op for every v-scoped agent #1103). - Reducing the fleet's API usage (The fleet and every session share one GitHub API quota (don-petry, 5,000/hour) and the fleet alone uses about that much: it ran out on 2026-10-10 and failed a workflow .github-private#2197).
Activity