Skip to content

Canary promotions fail at tag-move step — _gh_move_tag swallows the API error (persistent since 2026-07-15) #743

Closed

Description

@don-petry

Summary

The Canary Rollout scheduled engine fails whenever a promotion is due: _gh_move_tag cannot move a <agent>/v<M>-<tier> channel tag, so the run exits 1. It is persistent (every promotion-due run since 2026-07-15 06:12Z) and blocks the whole release pipeline — agents no longer advance rings.

Not a consumer-facing outage: all channel tags resolve and consumer workflows run normally. The impact is stalled promotions only.

Symptoms (run Canary Rollout #29406566440, 10:00Z 2026-07-15)

advancing agent-shield/v2-ring0 -> 12b0075a9c48 on petry-projects/.github
##[error]failed to move agent-shield/v2-ring0 -> 12b0075a9c48 on petry-projects/.github
##[warning]promote of agent-shield returned 1 (continuing fleet)
... (auto-rebase, dependabot-automerge, dependency-audit — 4 moves, all failed)
##[error]Process completed with exit code 1
  • Last successful move: run #29284341779 (2026-07-13 20:55Z).
  • Intervening runs succeeded only because no promotion was due (0 move attempts; all gates BLOCKED/SOAKING). 06:12Z was the first promotion-due run and it failed → the move break was latent until exercised.
  • The _run_json: failed to fetch run list lines are a separate, pre-existing, non-fatal issue (present in the 02:11Z run which succeeded). Not the cause here.

Root-cause blocker: the error is suppressed

scripts/canary-rollout.sh → _gh_move_tag:

_gh_move_tag() {
  [ $# -lt 3 ] && return 1
  local repo="$1" tag="$2" sha="$3"
  gh api -X PATCH "repos/$repo/git/refs/tags/$tag" \
      -f sha="$sha" -F force=true >/dev/null 2>&1 && return 0   # <-- swallows the real error
  gh api -X POST "repos/$repo/git/refs" \
      -f ref="refs/tags/$tag" -f sha="$sha" >/dev/null 2>&1      # POST then 422s (ref exists)
}

Both gh api calls discard stderr, so the log only shows the generic "failed to move" — the actual GitHub API rejection is invisible.

What has been RULED OUT (diagnosed 2026-07-15)

Ask (two parts)

  1. Diagnostic (do first): stop suppressing the error in _gh_move_tag — capture the PATCH response and echo/::error:: it (keep the POST fallback but only when PATCH is a genuine 404/"not found", not for every failure). The very next promotion-due run will then print the real rejection, which should make the cause obvious.
  2. Fix whatever that reveals. Leading hypotheses to check once the error is visible: an App-token-vs-bypass recognition nuance on PATCH .../git/refs/tags/<protected> (admin bypass works, App bypass apparently doesn't right now), or a GitHub-side change to how Integration bypass is evaluated for tag updates.

Acceptance criteria

  • _gh_move_tag surfaces the underlying API error on failure (no silent 2>/dev/null on the operative call); a bats/regression test covers the failure path emitting the error.
  • A promotion-due Canary Rollout run advances a channel tag successfully (or the run clearly reports the real API reason if it's a GitHub-side/permission issue to escalate).
  • No change to the immutable vX.Y.Z release tags or the refs/tags/** ruleset without explicit maintainer sign-off.

Evidence pointers

  • Failing runs: #29393518224 (06:12Z), #29406566440 (10:00Z). Last good: #29284341779 (07-13 20:55Z).
  • Ruleset: release-channel-tags #18516227. App: petry-projects-release-manager (4193127).

Activity

  1. added
    dev-leadFor dev-lead agent pickup
    and removed
    dev-leadFor dev-lead agent pickup
    on Jul 15, 2026
  2. don-petry commented on Jul 15, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead Implementation Plan

    Issue: #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error

    Scope

    Stop _gh_move_tag from discarding the PATCH stderr: capture and surface the underlying GitHub API rejection via ::error::, and only fall back to the POST (create-ref) path on a genuine 404/"not found" so a non-404 rejection is no longer masked by a subsequent "ref already exists" 422.

    Implementation Checklist

    • scripts/canary-rollout.sh — rewrite _gh_move_tag: capture PATCH stdout+stderr, return 0 on success; on failure emit the real API message as a single-line ::error::; POST-fallback ONLY when PATCH is a genuine 404/not-found; surface POST errors too. No silent 2>/dev/null on the operative call.
    • No change to immutable vX.Y.Z release tags or the refs/tags/** ruleset (per acceptance criteria).

    Tests to Write

    • tests/canary_rollout.bats — non-404 PATCH rejection: _gh_move_tag returns non-zero, emits the API error, and does NOT attempt POST (no masking).
    • tests/canary_rollout.bats — genuine 404: falls back to POST (create) and succeeds.
    • tests/canary_rollout.bats — POST fallback failure also surfaces its API error.
    • tests/canary_rollout.bats — successful PATCH moves the tag and never POSTs.

    Test Command

    node_modules/.bin/bats tests/canary_rollout.bats + bash .dev-lead/scripts/dev-lead-lint.sh

  3. don-petry commented on Jul 15, 2026

    @don-petry
    ContributorAuthor

    Dev-Lead: Implementation Complete

    Plan Execution

    • scripts/canary-rollout.sh — _gh_move_tag no longer swallows the PATCH error; captures stdout+stderr and surfaces the real API rejection as a single-line ::error:: on failure.
    • POST create-ref fallback is now taken only on a genuine 404/"not found" — a non-404 rejection is reported instead of being masked by a subsequent 422 "Reference already exists".
    • tests/canary_rollout.bats — 4 regression tests covering the failure/success/fallback paths.
    • No change to immutable vX.Y.Z release tags or the refs/tags/** ruleset.

    Test Results

    Full suite: 215/215 passing. New _gh_move_tag tests:

    1..4
    ok 1 _gh_move_tag: surfaces the API rejection and does NOT fall back to POST on a non-404 PATCH failure (#743)
    ok 2 _gh_move_tag: a successful PATCH moves the tag and never falls back to POST (#743)
    ok 3 _gh_move_tag: falls back to POST (create) when the ref is genuinely absent (404) (#743)
    ok 4 _gh_move_tag: surfaces the create error when the 404 POST fallback also fails (#743)
    

    Lint (.dev-lead/scripts/dev-lead-lint.sh): shellcheck (23 files) + frontmatter + CODEOWNERS — all checks passed.

    Files Changed

    • scripts/canary-rollout.sh: rewrote _gh_move_tag to capture the PATCH response (2>&1 into a var, not >/dev/null 2>&1), return 0 on success, emit ::error:: with the underlying API reason on failure, and gate the POST fallback on a genuine 404.
    • tests/canary_rollout.bats: added a _move_tag_stub helper + 4 tests — (1) non-404 PATCH rejection surfaces the error and does NOT POST; (2) successful PATCH never POSTs; (3) genuine 404 falls back to POST; (4) a failing 404-POST fallback surfaces the create error.

    Notes

    • Part 1 (diagnostic) is complete and verified. The next promotion-due Canary Rollout run will now print the real GitHub API rejection instead of only the generic "failed to move".
    • Part 2 (the underlying fix): the leading hypothesis — an App-token-vs-bypass recognition nuance on PATCH .../git/refs/tags/<protected> — is a GitHub-side/permission condition that can only be observed from a live run's newly-surfaced error. Per the acceptance criteria ("…or the run clearly reports the real API reason if it's a GitHub-side/permission issue to escalate"), this change delivers that visibility; the concrete remediation/escalation should follow the first post-merge promotion-due run's now-visible error.
    • The existing caller-side ::error::failed to move … lines are intentionally kept — they carry agent/ring/sha context, and now pair with the inner API-reason line.
  4. added a commit that references this issue on Jul 15, 2026
  5. added a commit that references this issue on Jul 16, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    dev-leadFor dev-lead agent pickup

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions