Skip to content

canary-rollout: tag resolution cannot tell an API error from an absent tag, so a total lookup outage reads as "fully rolled out" #1225

Closed

Description

@don-petry

Summary

canary-rollout resolves each ring's commit with _gh_tag_commit (scripts/canary-rollout.sh), which returns empty for any failure: a genuinely absent tag (404), a 5xx, a rate limit, a network error. Callers cannot tell them apart.

When every ring lookup comes back empty, all adjacent rings compare equal ("" == ""), so _frontier_state emits a single COMPLETE line ("fully rolled out"). During a GitHub API outage that can:

  • report an in-flight rollout as complete,
  • cause cmd_sync_issues to close an open canary-blocker / canary-confirm issue as "gate cleared", and
  • make promote-all report "nothing to promote".

Run-history fetch failures already fail closed (_frontier_state_resilient, #820). Tag-resolution failures do not.

Context

Raised independently by CodeAnt and cubic (twice) while reviewing #1220 / #1222 (#1118, per-pair evaluation). It was deliberately not changed there:

So the remaining gap is exactly the case that cannot be fixed without distinguishing the two causes.

Proposed change

  1. Make _gh_tag_commit report why it returned empty: absent (404 / no ref) vs. error (anything else). For example, a distinct exit status or a flag file, in the style of the existing _CANARY_FETCH_FAIL_FLAG for run history.
  2. In _frontier_state (and _frontier_state_resilient), a tag lookup that errored is unresolved-and-unknown: that pair is held BLOCKED (fail closed), and sync-issues must not close an existing blocker on the strength of it.
  3. A lookup that is absent keeps today's behavior, so unseeded and legacy agents are unaffected (AC6 of canary-rollout evaluates only the newest candidate — every new cut cancels a pending ring1→stable human confirmation, so dev-lead stable is frozen at 09-07 #1118 holds).
  4. Local-git agents (git rev-parse) have no API failure mode, so no change there.

Acceptance criteria

  • A total tag-lookup outage (every gh api .../git/ref/tags/... errors) yields BLOCKED / data-gap state and never COMPLETE; sync-issues fails closed (non-zero), as for a run-history outage.
  • An agent whose tags are all genuinely absent (404) still reports the legacy COMPLETE.
  • A single errored lookup among resolved ones holds only the affected pair.
  • Tests cover all three, using the existing gh stub pattern; shellcheck --severity=warning -x clean.

Notes

  • Touches tag resolution for every agent, so it deserves its own review rather than riding on a gate change. This is the release gate for the whole fleet: fail closed.
  • Related but separate, not covered here: forced tag writes in cmd_promote are not revalidated against the pair snapshot, so a scheduled sweep overlapping a manual per-agent dispatch could overwrite a newer tag. Workflow concurrency groups bound this today. A compare-and-swap on the write would harden it.

Refs #1118, #1176, #820.

Activity

  1. don-petry commented on Oct 3, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #1225 — canary-rollout: tag resolution cannot tell an API error from an absent tag, so a total lookup outage reads as "fully rolled out"

    Scope

    Make _gh_tag_commit distinguish an absent tag (404) from a lookup error, and fail closed on errored lookups in _frontier_state, _frontier_state_resilient and sync-issues, leaving the legacy all-absent → COMPLETE behaviour unchanged.

    Implementation Checklist

    • scripts/canary-rollout.sh _gh_tag_commit: capture stderr; 404 / "not found" → empty (absent, as today); any other failure (including a failed annotated-tag deref) → empty plus ::warning:: plus a record appended to $_CANARY_TAG_FAIL_FLAG when armed (same pattern as _CANARY_FETCH_FAIL_FLAG). Still never fails the caller, so the other call sites keep working.
    • New _ring_commits <agent> helper: resolves each ring once and reports <channel> <commit|-> <unknown 0|1> per ring, using a per-ring flag file.
    • _frontier_state: any pair touching a ring whose lookup errored is held BLOCKED (fail closed) without running _pair_state. When _CANARY_TAG_GAP_FLAG is armed it records that transition. Absent tags compare as before, so all-404 still gives COMPLETE.
    • _frontier_state_resilient: arm _CANARY_TAG_GAP_FLAG and tag the held pairs with datagap=2 (tag lookup failed). The run-history data-gap reconstruction also uses _ring_commits, so unknown rings are held with datagap=2.
    • cmd_sync_issues: on any datagap=2 pair, upsert the blocker with tag-gap evidence/title/banner, close neither the blocker nor stale confirm issues, emit ::error::, and exit non-zero (fail closed, like a run-history outage).
    • _blocker_body: tag-gap banner for data_gap=2. cmd_evaluate indeterminate message: also mention unresolved ring tags.

    Tests to Write (tests/canary_rollout.bats, _multicand_stub gh-stub pattern)

    • Total tag-lookup outage (every git/ref/tags/... → HTTP 502) → evaluate shows BLOCKED and never "fully rolled out". sync-issues exits non-zero and does not close an existing open blocker.
    • All tags genuinely absent (HTTP 404) → legacy COMPLETE / "fully rolled out", and sync-issues exits 0.
    • One errored lookup among resolved ones → only the pair(s) touching that ring are held, and other pairs evaluate normally.
    • _gh_tag_commit unit: 404 → empty and no flag; 5xx → empty and flag recorded.
    • Fix the _multicand_stub NONE fixture to model a real absent tag (stderr HTTP 404, exit 1) rather than a silent non-zero exit.

    Test Command

    bats tests/canary_rollout.bats (plus bash .dev-lead/scripts/dev-lead-lint.sh)

  2. don-petry commented on Oct 3, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete — PR #1240

    The implementation for issue #1225 is durable: commits are pushed and a pull request is open. This record is backed by verifiable artifacts you can confirm in one click.

    Review happens on the PR. Acceptance criteria and test results are evidenced by the PR's diff and CI checks — not asserted here. If a later stage fails, this claim is retracted in place.

  3. added a commit that references this issue on Oct 3, 2026
  4. added 2 commits that reference this issue on Oct 3, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions