Repository navigation
Description
Summary
canary-rollout resolves each ring's commit with _gh_tag_commit (scripts/canary-rollout.sh), which returns empty for any failure: a genuinely absent tag (404), a 5xx, a rate limit, a network error. Callers cannot tell them apart.
When every ring lookup comes back empty, all adjacent rings compare equal ("" == ""), so _frontier_state emits a single COMPLETE line ("fully rolled out"). During a GitHub API outage that can:
- report an in-flight rollout as complete,
- cause
cmd_sync_issuesto close an opencanary-blocker/canary-confirmissue as "gate cleared", and - make
promote-allreport "nothing to promote".
Run-history fetch failures already fail closed (_frontier_state_resilient, #820). Tag-resolution failures do not.
Context
Raised independently by CodeAnt and cubic (twice) while reviewing #1220 / #1222 (#1118, per-pair evaluation). It was deliberately not changed there:
- The "all lookups empty → COMPLETE" result is pre-existing: the single-frontier code treated an empty
nextplus empty rings as equal in the same way. - canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation, so dev-lead stable is frozen at 09-07 #1118 AC6 requires legacy bare-only and not-yet-seeded agents (no channel tags yet) to keep that behavior, so simply making all-empty BLOCKED would open blocker issues for every registered agent that has no tags.
- A partial failure is already fail-closed after canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation, so dev-lead stable is frozen at 09-07 #1118: an empty source commit with a populated destination is a pending BLOCKED pair in
_frontier_stateand in the resilient path (tested).
So the remaining gap is exactly the case that cannot be fixed without distinguishing the two causes.
Proposed change
- Make
_gh_tag_commitreport why it returned empty: absent (404 / no ref) vs. error (anything else). For example, a distinct exit status or a flag file, in the style of the existing_CANARY_FETCH_FAIL_FLAGfor run history. - In
_frontier_state(and_frontier_state_resilient), a tag lookup that errored is unresolved-and-unknown: that pair is held BLOCKED (fail closed), andsync-issuesmust not close an existing blocker on the strength of it. - A lookup that is absent keeps today's behavior, so unseeded and legacy agents are unaffected (AC6 of canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation, so dev-lead stable is frozen at 09-07 #1118 holds).
- Local-git agents (
git rev-parse) have no API failure mode, so no change there.
Acceptance criteria
- A total tag-lookup outage (every
gh api .../git/ref/tags/...errors) yields BLOCKED / data-gap state and neverCOMPLETE;sync-issuesfails closed (non-zero), as for a run-history outage. - An agent whose tags are all genuinely absent (404) still reports the legacy
COMPLETE. - A single errored lookup among resolved ones holds only the affected pair.
- Tests cover all three, using the existing
ghstub pattern;shellcheck --severity=warning -xclean.
Notes
- Touches tag resolution for every agent, so it deserves its own review rather than riding on a gate change. This is the release gate for the whole fleet: fail closed.
- Related but separate, not covered here: forced tag writes in
cmd_promoteare not revalidated against the pair snapshot, so a scheduled sweep overlapping a manual per-agent dispatch could overwrite a newer tag. Workflowconcurrencygroups bound this today. A compare-and-swap on the write would harden it.
Activity
don-petry commentedon Oct 3, 2026
Dev-Lead Implementation Plan
Issue: #1225 — canary-rollout: tag resolution cannot tell an API error from an absent tag, so a total lookup outage reads as "fully rolled out"
Scope
Make
_gh_tag_commitdistinguish an absent tag (404) from a lookup error, and fail closed on errored lookups in_frontier_state,_frontier_state_resilientandsync-issues, leaving the legacy all-absent →COMPLETEbehaviour unchanged.Implementation Checklist
scripts/canary-rollout.sh_gh_tag_commit: capture stderr; 404 / "not found" → empty (absent, as today); any other failure (including a failed annotated-tag deref) → empty plus::warning::plus a record appended to$_CANARY_TAG_FAIL_FLAGwhen armed (same pattern as_CANARY_FETCH_FAIL_FLAG). Still never fails the caller, so the other call sites keep working._ring_commits <agent>helper: resolves each ring once and reports<channel> <commit|-> <unknown 0|1>per ring, using a per-ring flag file._frontier_state: any pair touching a ring whose lookup errored is held BLOCKED (fail closed) without running_pair_state. When_CANARY_TAG_GAP_FLAGis armed it records that transition. Absent tags compare as before, so all-404 still givesCOMPLETE._frontier_state_resilient: arm_CANARY_TAG_GAP_FLAGand tag the held pairs with datagap=2(tag lookup failed). The run-history data-gap reconstruction also uses_ring_commits, so unknown rings are held with datagap=2.cmd_sync_issues: on any datagap=2 pair, upsert the blocker with tag-gap evidence/title/banner, close neither the blocker nor stale confirm issues, emit::error::, and exit non-zero (fail closed, like a run-history outage)._blocker_body: tag-gap banner for data_gap=2.cmd_evaluateindeterminate message: also mention unresolved ring tags.Tests to Write (
tests/canary_rollout.bats,_multicand_stubgh-stub pattern)git/ref/tags/...→ HTTP 502) → evaluate shows BLOCKED and never "fully rolled out".sync-issuesexits non-zero and does not close an existing open blocker.COMPLETE/ "fully rolled out", and sync-issues exits 0._gh_tag_commitunit: 404 → empty and no flag; 5xx → empty and flag recorded._multicand_stubNONEfixture to model a real absent tag (stderrHTTP 404, exit 1) rather than a silent non-zero exit.Test Command
bats tests/canary_rollout.bats(plusbash .dev-lead/scripts/dev-lead-lint.sh)don-petry commentedon Oct 3, 2026
Dev-Lead: Implementation Complete — PR #1240
The implementation for issue #1225 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.
3328e0b3343f40ed45ee561f052c26c4a96634a9main: 2 file(s), +278/-41 linesReview happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.
feat: implement issue #1225 — canary-rollout: tag resolution cannot t…
Merge main (#1240 tag-gap hold) into dev-lead/issue-1224
Merge main into dev-lead/issue-1242-20261003-1639