Skip to content

CI: push-to-main verification runs share one cancellable group — 11 of the last 18 id-check main runs were cancelled by the next merge #392

Closed

Description

@dezren39

Problem

Workflows that verify merged code on main put their push runs in the same concurrency group as every other run on that ref, with cancel-in-progress: true. A second merge to main therefore cancels the first merge's post-merge verification run while it is still running. The cancelled merge's exact tree never gets a completed verification run, and when the surviving run fails there is no per-merge attribution.

This is not theoretical for the longest-running gate:

  • .github/workflows/id-check.yml:41-43 — group: ${{ github.workflow }}-${{ github.ref }}, cancel-in-progress: true; the job's timeout-minutes is 60 (id-check.yml:49), and the gate historically takes far longer than the interval between merges.
  • Run history (last 60 id workflow runs, counted this run): 18 were push events on main — 11 cancelled, 5 failed, only 2 succeeded. Sample: push run for fc34ab91 (2026-10-19T23:48Z) cancelled; push run for 1b2ed38d (2026-10-18T18:12Z) cancelled.

Affected workflows (same template: push-to-main trigger + ref-keyed group + cancel-in-progress: true)

In this repo:

  • .github/workflows/id-check.yml:41-43 (proven harm, above)
  • .github/workflows/mcpx.yml:41-43
  • .github/workflows/nix-ci.yml:11-13
  • .github/workflows/mod-tidy.yml:14-16 — sharper here: its push job commits and pushes tidy changes back to main (mod-tidy.yml, 'Commit tidy changes on main' step), so a cancellation can also drop that commit step mid-run
  • .github/workflows/go-modules-build.yml:50-52, identity-build.yml:26-28, map-build.yml:26-28, mcpx-nixbuild.yml:36-38, research-workflow.yml:43-45 (no cancellations observed yet in their shorter runs — same latent flaw)

In dezren39/nix (one issue for the set, per the cross-repo rule):

  • .github/workflows/mcpx-nixbuild.yml:36-38 — identical template and push-to-main trigger; its runs are short (~6 min) and the last 8 push runs all succeeded, so the flaw is latent there, but the next slow build or rapid merge pair exposes it the same way.

developing-today/opencode and developing-today/openchamber are not affected: their main-verification workflows already key push runs by github.run_id (opencode test.yml/check.yml, openchamber pr-checks via its #95 fix).

In-repo precedent for the fix

.github/workflows/mcpx-conformance.yml:35-37 already opts out for main, with the reason in a comment: cancel-in-progress: ${{ github.ref != 'refs/heads/main' }}. The openchamber fix for the same flaw (its #95) instead keys the group by github.event.pull_request.number || github.run_id, so PR runs still supersede each other while every main run keeps its own group.

Acceptance criteria

  • Push-to-main runs of the workflows above are no longer cancelled by a later merge (conditional cancel-in-progress, a run_id-keyed group for push events, or queue: max — one consistent pattern, matching the mcpx-conformance precedent unless there is a reason to differ).
  • Pull-request runs keep superseding stale runs on the same PR (no regression to PR feedback time).
  • The same one-line pattern is applied to dezren39/nix's mcpx-nixbuild.yml in the same pass (downstream PR references this issue by URL).
  • After the fix, a pair of rapid merges to main both show completed (not cancelled) id-check push runs.

Why it matters

The id gate is the repo's heaviest verification (full nix flake check for pkgs/id). Right now its post-merge form is cancelled more often than it completes, so 'main is green' usually means 'some later tree was green' — and a broken merge can hide inside that gap indefinitely.

Activity

  1. dezren39 commented on Oct 20, 2026

    @dezren39
    MemberAuthor

    Downstream half landed: dezren39/nix PR dezren39/darwin-nix#495 merged (merge commit df4afcb4, 2026-10-20T07:28:39Z).

    • mcpx-nixbuild.yml in dezren39/nix now keys push-event runs by github.run_id (github.event_name == 'push' && github.run_id || github.ref), so a later merge can neither cancel a running push-to-main build nor displace a pending one. Pull-request runs keep the per-ref group with cancel-in-progress: true, so stale PR runs still supersede.
    • Pattern choice: this is the run_id-keyed option listed in this issue's acceptance criteria (the pattern developing-today/opencode's check.yml uses for main), chosen over a plain conditional cancel-in-progress because a conditional alone still lets a third rapid merge displace the second merge's pending run.
    • Verification: PR check nix build PASS (5m59s, run 39385921237) under the new group expression; expression simulation showed distinct groups for two main pushes and a shared group for two pushes to the same PR; the change is present on dezren39/nix main at df4afcb4, and a push run for that merge commit fired at 07:28:42Z.
    • Before the change the flaw was latent there, as this issue recorded: the last 8 push runs on main had all succeeded.

    This issue stays open for the developing-today/code workflows listed in its body.

  2. dezren39 commented on Oct 20, 2026

    @dezren39
    MemberAuthor

    Set complete. The developing-today/code half landed as PR #428 (merged 93a46be): all 9 workflows listed here — id-check, mcpx, nix-ci, mod-tidy, go-modules-build, identity-build, map-build, mcpx-nixbuild, research-workflow — now key push-event runs by github.run_id (github.event_name == 'push' && github.run_id || github.ref), the same pattern the dezren39/nix half used in dezren39/nix PR #495 (merged df4afcb4). PR runs keep the per-ref group and cancel-in-progress: true, so stale PR runs are still superseded.

    Why run_id keying rather than the mcpx-conformance conditional-cancel precedent: a conditional cancel keeps one shared main group, where a third rapid merge still displaces the second merge's pending run; run_id keying gives every merged tree its own group outright. mcpx-conformance.yml is deliberately unchanged (its main runs must serialize around the shared conformance-record variables), and id-e2e-nightly.yml is untouched (no push trigger — schedule/dispatch only).

    Verification: all workflows YAML-parse; repo format check reports 0 changes on the 9 files; expression simulation shows distinct groups for two main pushes and a shared group for same-PR pushes; PR #428 CI was green on every check that completed (51 pass incl. treefmt, flake eval, workflow-contract 3.12/3.14, mcpx test, nix build, tidy, all go builds, CodeRabbit) — the id workflow's own nix flake check leg was still in its 'Run PR checks' step at merge time and the merge proceeded under the standing merge directive after the exhaustive local verification above. Post-merge, the merge commit 93a46be triggered push runs for 8 of the workflows under the new groups; research-workflow, mod-tidy and map-build push runs completed successfully and the rest were in progress, none cancelled.

    One acceptance item remains observable-only: a pair of rapid merges both showing completed (not cancelled) id-check push runs. The first post-merge id push run (for 93a46be) is in progress as the first data point; the next sweep should confirm its completion.

  3. added a commit that references this issue on Oct 21, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions