Repository navigation
Closed
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickup
Description
Summary
The Canary Rollout scheduled engine fails whenever a promotion is due: _gh_move_tag cannot move a <agent>/v<M>-<tier> channel tag, so the run exits 1. It is persistent (every promotion-due run since 2026-07-15 06:12Z) and blocks the whole release pipeline — agents no longer advance rings.
Not a consumer-facing outage: all channel tags resolve and consumer workflows run normally. The impact is stalled promotions only.
Symptoms (run Canary Rollout #29406566440, 10:00Z 2026-07-15)
advancing agent-shield/v2-ring0 -> 12b0075a9c48 on petry-projects/.github
##[error]failed to move agent-shield/v2-ring0 -> 12b0075a9c48 on petry-projects/.github
##[warning]promote of agent-shield returned 1 (continuing fleet)
... (auto-rebase, dependabot-automerge, dependency-audit — 4 moves, all failed)
##[error]Process completed with exit code 1
- Last successful move: run #29284341779 (2026-07-13 20:55Z).
- Intervening runs succeeded only because no promotion was due (0 move attempts; all gates BLOCKED/SOAKING). 06:12Z was the first promotion-due run and it failed → the move break was latent until exercised.
- The
_run_json: failed to fetch run listlines are a separate, pre-existing, non-fatal issue (present in the 02:11Z run which succeeded). Not the cause here.
Root-cause blocker: the error is suppressed
scripts/canary-rollout.sh → _gh_move_tag:
_gh_move_tag() {
[ $# -lt 3 ] && return 1
local repo="$1" tag="$2" sha="$3"
gh api -X PATCH "repos/$repo/git/refs/tags/$tag" \
-f sha="$sha" -F force=true >/dev/null 2>&1 && return 0 # <-- swallows the real error
gh api -X POST "repos/$repo/git/refs" \
-f ref="refs/tags/$tag" -f sha="$sha" >/dev/null 2>&1 # POST then 422s (ref exists)
}Both gh api calls discard stderr, so the log only shows the generic "failed to move" — the actual GitHub API rejection is invisible.
What has been RULED OUT (diagnosed 2026-07-15)
- NOT the bare-tag retirement/restore. The failing moves are on
v<M>-ring0tags that were never touched; consumers resolve; a manual admin-token no-op PATCH ofauto-rebase/v2-ring0to its own sha succeeded (200) — so the API, ruleset, and tag state are healthy. - NOT the App / ruleset / permissions. The engine mints
release-manager(app_id 4193127, client-idIv23liaWBjWQlzln9FGJ), which is a bypass actor on therelease-channel-tagsruleset (#18516227, rules: update+deletion; bypass: OrgAdmin + 3167543 + 4193127), hascontents:write, is not suspended, and is correctly wired asGH_TOKENinto the "Run canary-rollout" step (workflow line 173). - NOT a recent code regression in the move/token path. feat: implement issue #732 — Canary blocker: ci-failure-analyst next->ring0 (cum_fail=0, -) #733/feat: implement issue #712 — canary #1083 pillar 2: autocut breaking-change detection (auto major bump from conventional-commit + workflow_call interface diff) #720/feat: implement issue #704 — F5 gap: deploy-standard-workflows exempts the meta-repos (.github/.github-private), so the major-scope migration missed their self-consumer stubs [#657] #706 touched gate/autocut logic, not
_gh_move_tag, the token env, or the PATCH. - The move target is a clean fast-forward (
compare b7e6b670...12b0075a→ ahead=90, behind=0) at a valid commit.
Ask (two parts)
- Diagnostic (do first): stop suppressing the error in
_gh_move_tag— capture the PATCH response andecho/::error::it (keep the POST fallback but only when PATCH is a genuine 404/"not found", not for every failure). The very next promotion-due run will then print the real rejection, which should make the cause obvious. - Fix whatever that reveals. Leading hypotheses to check once the error is visible: an App-token-vs-bypass recognition nuance on
PATCH .../git/refs/tags/<protected>(admin bypass works, App bypass apparently doesn't right now), or a GitHub-side change to how Integration bypass is evaluated for tag updates.
Acceptance criteria
_gh_move_tagsurfaces the underlying API error on failure (no silent2>/dev/nullon the operative call); a bats/regression test covers the failure path emitting the error.- A promotion-due
Canary Rolloutrun advances a channel tag successfully (or the run clearly reports the real API reason if it's a GitHub-side/permission issue to escalate). - No change to the immutable
vX.Y.Zrelease tags or therefs/tags/**ruleset without explicit maintainer sign-off.
Evidence pointers
- Failing runs: #29393518224 (06:12Z), #29406566440 (10:00Z). Last good: #29284341779 (07-13 20:55Z).
- Ruleset:
release-channel-tags#18516227. App:petry-projects-release-manager(4193127).
Metadata
Metadata
Assignees
Labels
dev-leadFor dev-lead agent pickupFor dev-lead agent pickup
Activity
don-petry commentedon Jul 15, 2026
Dev-Lead Implementation Plan
Issue: #743 — Canary promotions fail at tag-move step — _gh_move_tag swallows the API error
Scope
Stop
_gh_move_tagfrom discarding the PATCH stderr: capture and surface the underlying GitHub API rejection via::error::, and only fall back to the POST (create-ref) path on a genuine 404/"not found" so a non-404 rejection is no longer masked by a subsequent "ref already exists" 422.Implementation Checklist
scripts/canary-rollout.sh— rewrite_gh_move_tag: capture PATCH stdout+stderr,return 0on success; on failure emit the real API message as a single-line::error::; POST-fallback ONLY when PATCH is a genuine 404/not-found; surface POST errors too. No silent2>/dev/nullon the operative call.vX.Y.Zrelease tags or therefs/tags/**ruleset (per acceptance criteria).Tests to Write
tests/canary_rollout.bats— non-404 PATCH rejection:_gh_move_tagreturns non-zero, emits the API error, and does NOT attempt POST (no masking).tests/canary_rollout.bats— genuine 404: falls back to POST (create) and succeeds.tests/canary_rollout.bats— POST fallback failure also surfaces its API error.tests/canary_rollout.bats— successful PATCH moves the tag and never POSTs.Test Command
node_modules/.bin/bats tests/canary_rollout.bats+bash .dev-lead/scripts/dev-lead-lint.shdon-petry commentedon Jul 15, 2026
Dev-Lead: Implementation Complete
Plan Execution
scripts/canary-rollout.sh—_gh_move_tagno longer swallows the PATCH error; captures stdout+stderr and surfaces the real API rejection as a single-line::error::on failure.422 "Reference already exists".tests/canary_rollout.bats— 4 regression tests covering the failure/success/fallback paths.vX.Y.Zrelease tags or therefs/tags/**ruleset.Test Results
Full suite: 215/215 passing. New
_gh_move_tagtests:Lint (
.dev-lead/scripts/dev-lead-lint.sh): shellcheck (23 files) + frontmatter + CODEOWNERS — all checks passed.Files Changed
scripts/canary-rollout.sh: rewrote_gh_move_tagto capture the PATCH response (2>&1into a var, not>/dev/null 2>&1),return 0on success, emit::error::with the underlying API reason on failure, and gate the POST fallback on a genuine 404.tests/canary_rollout.bats: added a_move_tag_stubhelper + 4 tests — (1) non-404 PATCH rejection surfaces the error and does NOT POST; (2) successful PATCH never POSTs; (3) genuine 404 falls back to POST; (4) a failing 404-POST fallback surfaces the create error.Notes
Canary Rolloutrun will now print the real GitHub API rejection instead of only the generic "failed to move".PATCH .../git/refs/tags/<protected>— is a GitHub-side/permission condition that can only be observed from a live run's newly-surfaced error. Per the acceptance criteria ("…or the run clearly reports the real API reason if it's a GitHub-side/permission issue to escalate"), this change delivers that visibility; the concrete remediation/escalation should follow the first post-merge promotion-due run's now-visible error.::error::failed to move …lines are intentionally kept — they carry agent/ring/sha context, and now pair with the inner API-reason line.feat: implement issue #743 — Canary promotions fail at tag-move step …
feat: implement issue #745 — Canary tag-move 403 'Resource not access…