Repository navigation
fix(ci): a half-finished release leaves a v* tag that makes every retry skip silently #888
Description
Activity
Checkable blocker: #887, and my recommendation for when it clears.
gh issue view 887 -R JarryShaw/PyPCAPKit --json state
#887 changes this defect's failure surface, so fixing it first would mean designing against a shape that is about to move. Today the
v*tag can be left behind two ways: a rejected approval on one of the four gates aftergithub-releasewas approved, or a failed job after tagging. Collapsing to one approval removes the first entirely — reject and nothing is tagged at all — leaving only the mid-run failure. A fix that handles both is more machinery than a fix that handles one.Recommendation, for when #887 lands: the first of the three shapes — gate each publish on its own evidence rather than on the tag. Skip the PyPI upload only when that version is already on PyPI, and Anaconda only when it is already on Anaconda. That makes each job self-healing on retry, which is the property actually wanted here: a half-finished release completes on the next run instead of being skipped or needing a hand-pushed tag. It also removes the coupling that makes
PCAPKIT_TAG_EXISTSdo work it was never suited for — the tag answers "was this tagged", not "was this published", and those are the same only while nothing fails.The second shape (make the skip explicit, failing or annotating when the tag exists and the trigger was not a
v*push) is worth doing regardless, and is a few lines. It does not fix the retry, but it converts a green no-op into something visible, which is the part that actually cost time here. If only one thing gets done, it should be this.Not recommending the third beyond documentation: pushing the
v<version>tag by hand works because it takes thestartsWith(github.ref_name, 'v')branch, but it is a workaround that depends on someone knowing this issue exists.Labelled
blockedon #887. Nothing dispatched.Revising my own recommendation above: the manual-tag recovery is withdrawn, and this issue is now the whole of the remaining gap.
The owner's ruling on #887 is "Only bump
pcapkit.__version__string. Never manual tag or touch anything." My note above recommended pushing thev<version>tag by hand to recover from a half-finished release. That is exactly what the ruling forbids, and the ruling is right on the mechanics, not just on hygiene: a hand-pushed tag takes thepush: tags: ['v*']trigger, wherestartsWith(github.ref_name, 'v')short-circuits the guard, so the workaround works precisely because it disables the check that would otherwise stop a double-publish. Withdrawn.What that leaves. #887 has landed its settings half — one approval on
github-release, reviewers cleared fromconda-tag,pypiandanaconda— so the rejected-gate route into a stranded tag is gone: reject and nothing is tagged at all. What survives is the route this issue is actually about: thegithubjob succeeds, the tag exists, and a later job fails. Then on theworkflow_runpath (github.ref_name=main, so thestartsWithshort-circuit does not apply) every release job is skipped and the run reports green, with no sanctioned way to retry.So the blocker is cleared and the shape has narrowed. My recommendation stands and firms up:
Gate each publish on its own evidence, not on the tag. Skip the PyPI upload only when that version is already on PyPI; skip Anaconda only when it is already there. Each job then self-heals on re-run, which is the only recovery compatible with "never manual tag" — you re-run the workflow and the jobs that did not finish simply finish. The tag answers "was this tagged", never "was this published", and those diverge exactly when something fails mid-release.
And the cheap half is now more valuable than before, not less: make the skip explicit — fail, or at minimum annotate, when the tag exists and the trigger was not a
v*push. Without it the operator's only signal that a release did not happen is a green run that did nothing, and with manual tagging correctly off the table there is no longer an informal escape hatch to notice the problem by.Removing
blocked; #887's settings change has landed and its remaining work (theneeds:fix and the release doc) does not gate this. Not dispatching yet — the doc being written for #887 has to describe this failure mode and its absence of a recovery, and I would rather the two land in an order where the doc does not immediately go stale. Ready to dispatch on your word, or once #887's PR is in.Checkable blocker: #893, which is in review now and documents this gap as it currently stands.
gh pr view 893 -R JarryShaw/PyPCAPKit --json state,mergedAt
#893 adds
docs/source/releasing.rst, whose recovery section says in terms that re-running is the sanctioned route, that it can silently no-op on theworkflow_runpath, and that no sanctioned manual recovery exists for an already-stranded tag — citing this issue as the reason. Changing that behaviour while #893 is under cross-review would invalidate the review of prose describing it, and land a document that is wrong on arrival.So this is sequenced, not idle: once #893 merges, this gets a worker and the doc gets the corresponding paragraph updated in the same change.
blockedapplied with that written down, since a board entry with no label reads as nobody's problem.Recommendation unchanged from above — gate each publish on its own evidence (already on PyPI / already on Anaconda) so a plain re-run self-heals, plus the cheap half of making the skip explicit rather than a green no-op.
You are right, and I over-corrected — twice in the same direction, which is worth naming.
Read #887 again, we had agreed that manual tagging should still be allowed and supported.
Agreed there and still true. My first body edit struck the recovery bullet through as though hand-tagging were the problem; it is not. Body now says what actually holds: pushing a fresh
v<version>tag is a supported way to start a release, and the thing that does not work is using a tag push to retry a stranded one — because in that case the tag already exists, so there is no new ref event to fire, and forcing one means deleting a ref the automation owns.So the sentence is "there is no sanctioned recovery for an already-stranded tag, not because hand-tagging is forbidden but because there is no fresh tag left to push". That is a statement about the situation, not a prohibition on the tool — which is the distinction I kept collapsing.
The cross-review of #893 found the same confusion had reached the release doc from a different angle, so the corrected framing is going into
releasing.rstin that PR's current round too.Unblocked — #893 merged as
486c2a269at 2026-09-29T02:07:23Z, so the blocker recorded above is discharged.blockedremoved.That leaves the one thing #893 could not settle: which of the two shapes in the body to build. They are not alternatives so much as different jobs, and my recommendation is both, in this order:
- Gate each publish on its own evidence — skip the PyPI upload only when that version is already on PyPI, and Anaconda only when it is already on Anaconda. This is the actual fix: it makes a retry after a partial release self-heal, and it stops using "tag exists" as a proxy for "this version shipped", which is precisely the inference that breaks when a release half-happens.
- Make a skip-because-already-released outcome explicit — annotate or fail rather than passing green. On its own this only stops the silent no-op and leaves you with no retry path, so it is the safety net, not the fix. Worth having because it covers the case where (1)'s evidence check is itself wrong.
The ask: both, or (2) alone for now? I am recommending both. (1) is the larger change and touches the jobs that publish to PyPI and Anaconda, so it is also the one where I would rather have your word than my judgement.
Marking
needs: decisionfor that choice only. Meanwhile I am dispatching a read-only investigation to establish the part that is the same either way — whether a version's presence on PyPI and on Anaconda can be checked reliably from inside the workflow, and what the existing jobs already have in hand. Nothing will be committed or pushed from it; it just means the implementation is ready the moment you rule.Also still open from the body, unrelated and not part of this:
tagrunsgit tag -d v<version> || trueat:275, which is local-only and deletes nothing on the remote, but reads like "delete the release tag".3 remaining items
Correction to this issue's own body, found by watching a live release:
version_checkis not ungated, and there is a second silent-skip path this issue does not describe.The body says "
version_checkis ungated and read-only". It is not. At currentorigin/main:96 unit-tests: 97 name: Release test gate 98 if: ${{ github.event_name != 'workflow_run' || github.event.workflow_run.conclusion == 'success' }} ... 105 version_check: 106 name: Check Version 108 needs: [ unit-tests ]So when a
workflow_runtrigger arrives from a Vendor Update run that concluded anything other thansuccess,unit-testsskips,version_checkskips with it, and every downstream release job skips — without the sharedPCAPKIT_TAG_EXISTSguard ever being evaluated.Observed, not theorised. Two
Create Releaseruns today concludedskippedwithCheck Versionitself skipped:run 36511610205 02:13 486c2a26 Check Version skipped, all 5 publish jobs skipped run 36510774776 02:03 5e7f91eb sameSo I was wrong to read those as this issue's defect. They are the other path: the triggering Vendor Update had nothing to do, so the release correctly did nothing. The run still reports
skippedrather than anything a human would notice, which is the same complaint — a green no-op — but a different mechanism, and conflating them would have produced a fix aimed at the wrong guard.This has a direct consequence for the fix now being implemented. Gating
pypi'sif:on an evidence output fromversion_checkinheritsversion_check's own skip: ifunit-testsskips, the output never exists andpypiskips regardless of what the evidence would have said. An evidence check that lives downstream of a skippable job cannot make a skip loud. Either the evidence job must not depend onunit-tests, or the "did this skip for a legitimate reason" annotation has to sit somewhere that always runs. I have sent this to the worker.Meanwhile the healthy path is working. The Vendor Update bot bumped to
1.5.0b6at 02:29:58Z (59dc8c0aa, also regenerating the fivepcapkit/const/reg/apptype/*.pyfiles), and a release is in progress now on that sha —Release test gate / Gate (full suite)running, run36512890404.v1.5.0b6does not exist yet (git ls-remote --tagsshows onlyv1.5.0b4andv1.5.0b5), soPCAPKIT_TAG_EXISTSisfalseand the guard passes as designed. Nothing for me to do there; flagging it so you know a release is live.One incidental:
v1.5.0b5points atae8a9467d, a docs-only commit of mine.I'm thinking we might want to include somewhere in the documentation for workflow graphs of these CI/CD GHA workflows.
Good ask, and it is broader than this issue, so I have given it its own: #897.
I'm thinking we might want to include somewhere in the documentation for workflow graphs of these CI/CD GHA workflows.
There are eight workflows and the interesting structure is the edges between them, which no single file shows:
deploy-pages.yml,create-release.ymlandcron-conda.ymlall fire onworkflow_runfrom an upstream workflow, andcreate-release.ymladditionally callsunit-tests.ymlas a reusable workflow — a different relationship from triggering it. That distinction is exactly what bit this issue: the skip cascade I posted about above runsunit-tests→version_check→ every publish job, and it is invisible from any one file.Scoped #897 to cover trigger kinds, the
workflow_runedges, reusable-workflow calls, where the required status checks come from, and which jobs declare anenvironment:and so can pause for approval.docs/source/releasing.rststays out of scope — it already documents the release path and is being edited under this issue right now; #897 will cross-reference it rather than duplicate it.Worker dispatched on #897. This issue keeps its own scope: the guard fix.
- added 4 commits that reference this issue
on Sep 29, 2026
Describe the bug
Every release job in
create-release.ymlshares one guard:version_check(:89-130) is ungated and read-only — it runsmukunku/tag-exists-action@v1.7.0againstv<version>and emitsPCAPKIT_TAG_EXISTS. Thev*tag itself is created bysoftprops/action-gh-releaseat:199(tag_nameat:216) inside thegithubjob. So once the tag exists but the publish did not complete, the guard's behaviour splits by trigger:github.ref_namestartsWith(…, 'v')push: tags: ['v*'](:2-5)v1.5.0workflow_run← "Vendor Update" (:6-9)main— the run evaluates the default-branch tip, per the comment at:15-19Reproduction
Not run against production — derived from the workflow at
ae8a9467d. The reachable sequence: aworkflow_run-triggered run tags and publishes the GitHub Release, thenpypiorcondafails (or, today, its separate approval is rejected — see #887). Thev<version>tag now exists with no PyPI/Anaconda upload. The nextworkflow_runrun readsPCAPKIT_TAG_EXISTS=true,ref_name=main, and every release job is skipped. The run is green, because skipped is not failed, so nothing surfaces.Expected behavior
A retry after a partial release either completes the missing publishes or fails loudly. "Tag exists" is being used as a proxy for "this version was already released", and those diverge exactly when a release half-happens — which is the case the guard most needs to handle.
Shapes worth considering, in rough order of cost:
v*push. Cheapest way to stop a green no-op.v<version>tag by hand is a supported way to start a release and always was — see ci(release): one approval for the whole release, not four separate environment gates #887, where that was settled explicitly. What does not work is using a tag push to retry a stranded one: in that scenario the tag already exists, and apushtrigger fires on ref creation or update, so re-pushing an identical ref produces no event. Making it fire would mean deleting and re-pushing, which touches a ref the release automation owns and lands the retry on thestartsWithpath with the tag-exists check bypassed, against a version that may already be on PyPI. So there is no sanctioned recovery for an already-stranded tag — not because hand-tagging is forbidden, but because there is no fresh tag left to push.System information
Workflow file only; no library version involved. Read at
origin/main=ae8a9467d.Additional context
Found while answering whether the manual approvals gate tagging (they do — the tag is created inside the
github-release-gated job). #887 carries the separate ask to collapse four approvals into one, which makes this rarer but does not remove it: a mid-run failure after tagging still lands here.One unrelated oddity noticed in the same trace, recorded so it is not lost: the
tagjob runsgit tag -d v<version> || trueat:275before creatingconda-<version>+0. It is local-only so it deletes nothing on the remote, but it reads like "delete the release tag" and cost a second look.