Skip to content
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .github/workflows/verify.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# The required check, `factory / verify`: it passes only when `make verify` and every shard of the mutation check pass on the commit, with the history tools/check_decisions.py compares it with. On a pull request the shards decline and pass, and on a merge group each runs in full (record 0182).
# `checks` runs `make verify` in three legs, its simulators on 300 seeds: each leg runs every check and every third test, told its place and the count by the matrix itself, never by a number written in, so that across the legs each test runs once. As one job, on 2026-10-21 the runs that finished took 11 to 14.5 minutes, and six runs in three pull requests were cut at its 15-minute timeout with the tests still running, so its timeout was raised to 25 minutes; by 2026-10-29 a run took up to 23, so the job was given 28 and its step 26, two kept for the checkout; and on 2026-11-09 the ten runs before took 19 to 23.4 and a merge group was cut at 26.2, so its tests were dealt into legs. Add a leg before one nears the step's limit.
# About 6,400 mutants, in sixteen shards of about 400, each kill confirmed by two runs of one test. With six shards of 168, of about 1,000 mutants then, the legs ran 14 to 25 minutes against a 25-minute timeout, and legs were cut at 25:00 in three pull requests, once 22 ms after the last kill had printed. With eight shards of 651 the legs mostly ran 6 to 14 minutes, and on 2026-11-06 one was cut at its step's 27 minutes on a runner that ran it 2.8 times slower than a re-run of the same tree ("CI: twelve mutant shards, so a slow runner does not cut a leg at its limit", issue 583); the first run of twelve took 4.4 to 12.3 minutes a leg, 7.3 the median. By 2026-11-10, with about 6,390 mutants, the ten merge groups before took 15.8 to 17.3 minutes at the slowest leg's job, and nine kept under 40% of the step's 27 to spare, so the shards became sixteen. Every mutant names the tests that kill it, which run alone ("Mutation run: every mutant names its tests, the slow suites run last, and a builder may break a few", issue 217); a mutant they miss is an ERROR, named_test_does_not_kill, since the whole suite run after them took 10 to 21 minutes of a leg (2026-11-09). Add a shard before a leg nears its timeout, or before a leg of the median time would take more than 22 minutes on a runner three times slower. A runner as slow may still cut the longest leg: run it again.
# A pull request's run is cancelled when a newer push to it starts another: on 2026-11-09, 13 of a day's 47 ran on after a newer push, about 870 runner-minutes on the Free plan's 20 runners. A merge group's run is never cancelled: its group is its own run's id, and it cancels nothing; nor does a re-run, whose attempt is past its first, which would otherwise cancel the run of a newer head. The cancelled run's failed verify is passed over beside a later run on the same head (decision 0203), and decision 0206 amends 0014's row. tools/check_workflows.py holds both lines word for word.
# `verify` checks out the base, never the head, and runs the base's tools/needs_passed.py, so a change cannot make its own verdict pass: decision 0023.
# `commits` checks out the base too and runs the base's tools/commits_are_the_maintainers.py, which reads every commit the pull request carries and refuses unless each names the Maintainer ("Identity: every commit a pull request carries is the Maintainer's, checked outside the shell", issue 204). It reads the event's payload too, and refuses unless an account on the configuration's list opened the pull request from this repo: decision 0102. On a merge group, whose payload names no opener, it reads the pull request the queue's ref names, with this job's read-only token.
name: factory
on:
pull_request:
merge_group:
permissions: {}
concurrency:
group: ${{ format('factory-pr-{0}', github.event_name == 'pull_request' && github.run_attempt == 1 && github.event.pull_request.number || github.run_id) }}
Comment thread
don-petry marked this conversation as resolved.
Resolved
cancel-in-progress: ${{ github.event_name == 'pull_request' && github.run_attempt == 1 }}
Comment thread
don-petry marked this conversation as resolved.
Resolved
jobs:
checks:
runs-on: ubuntu-24.04
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,27 @@
# 0206. The required check's pull-request run gives way to a newer push's, and a merge group's run is never cancelled

Status: accepted. The Factory Builder's rulings of 2026-11-09 and 2026-11-13 on issue 681, a Lens's cancelled run beside a later one on the same head: its plan's step 4, the cancel of a pull request's run a newer push outdates, with the points of the Claude review of "CI: a newer push cancels its pull request's verify run, and a merge group's run is never cancelled" (pull request 679). It amends decision 0014's row "A concurrency group on CI", where it says none on the required check; the rest of the row stands.
Comment thread
don-petry marked this conversation as resolved.
Resolved

## Decision

| What | Decided | Because |
|---|---|---|
| The group | `.github/workflows/verify.yml` has one top-level concurrency: its group `${{ format('factory-pr-{0}', github.event_name == 'pull_request' && github.run_attempt == 1 && github.event.pull_request.number \|\| github.run_id) }}` and its `cancel-in-progress` `${{ github.event_name == 'pull_request' && github.run_attempt == 1 }}`. A pull request's run gives way to the run a newer push to the same pull request starts | On 2026-11-09, 13 of a day's 47 pull-request runs ran on after a newer push: about 870 runner-minutes, 14% of that day's pull-request runner time, on the Free plan's 20 runners |
| A merge group's run | Never cancelled, and it cancels nothing: its group is its own run's id, unlike any other's | A cancelled merge group's run would eject its entries from the queue |
| A re-run | Cancels nothing and replaces nothing: its attempt is past its first, so its group is its own run's id, as a merge group's is | A re-run keeps its `pull_request` event and number, so in the pull request's group a re-run of an older head's run would cancel the newest head's run in progress (Astra), or take the place of the one waiting, since a group keeps one run waiting whatever `cancel-in-progress` says (cubic and CodeAnt, reading this record's change); no later run on that head replaces either |
| Why 0014's reason no longer forbids it | 0014 kept the required check out of any group because a group evicts a waiting run, which once lost work (`docs/lessons.md`, "A request existed only as a queued run"). A pull request's run is no request: the run a newer push starts judges the newer head, and what is evicted judged content no longer the head. Each Run's lane keeps its own group, which never cancels the Run in progress | The lesson stands for every workflow that carries a request |
| The cancelled run's check | A cancelled run's `verify` concludes `failure` under `if: always()`, its suite `cancelled` (`trials/P-65.json`). On the head it ran on, beside a later run of the file, it is passed over (decision 0203); on an older head nothing reads it | Step 3 lands first, so the cancel cannot fail a Lens on content a later run passed |
| Only the required check's workflow | `tools/check_workflows.py` allows the two expressions in `.github/workflows/verify.yml`'s top-level concurrency alone, and names them in its rule `expression`; in any other workflow each is an expression where text from outside could stand | The Claude review of "CI: a newer push cancels its pull request's verify run, and a merge group's run is never cancelled" (pull request 679) |
| What remains | The secret scan's `.github/workflows/ci.yml` has no group, so its superseded runs still run to the end. A person's cancel and a reopening each leave a cancelled run beside a later one on the same head, which decision 0203 reads. The group's own cancel of a run in progress, and its eviction of a run still waiting, which may start no job and leave no `verify` at all, were not tried: P-65 cancelled by hand; either is of a head a newer push replaced | Named; the secret scan's runs take under a minute |

## Why

Decision 0014 kept the required check out of any concurrency group for a reason that holds for a request and not for a
pull request's run. Once decision 0203 reads a cancelled run beside a later one, the run a newer push outdates can give
way, and only the merge group's run, which the queue needs whole, is kept out of every group but its own.

## Boundaries

| Boundary | Check |
|---|---|
| The required check's workflow holds that concurrency, or none, and no other; a merge group's run and a re-run are never cancelled nor cancel another's; the two expressions stand there alone, and in any other workflow, each alone too, are refused | `tests/test_tools.py::test_only_a_superseded_pull_request_run_is_cancelled`, `tests/test_tools.py::test_a_rule_with_no_counter_example_is_caught`, `tools/check_workflows.py`, mutant: "the required check's concurrency may be anything", mutant: "a pull request's lane is no expression the check allows", mutant: "any top-level concurrency expression is allowed", mutant: "a pull request's lane stands in any workflow", mutant: "the cancel stands in any workflow", mutant: "every workflow is linted as the required check's", mutant: "the required check's workflow is linted as any other" |
6 changes: 5 additions & 1 deletion docs/standards/workflows.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,13 +12,17 @@ Any other spelling is refused. An org standard is cited from combsmith/.github a
# The required check, `factory / verify`: it passes only when `make verify` and every shard of the mutation check pass on the commit, with the history tools/check_decisions.py compares it with. On a pull request the shards decline and pass, and on a merge group each runs in full (record 0182).
# `checks` runs `make verify` in three legs, its simulators on 300 seeds: each leg runs every check and every third test, told its place and the count by the matrix itself, never by a number written in, so that across the legs each test runs once. As one job, on 2026-10-21 the runs that finished took 11 to 14.5 minutes, and six runs in three pull requests were cut at its 15-minute timeout with the tests still running, so its timeout was raised to 25 minutes; by 2026-10-29 a run took up to 23, so the job was given 28 and its step 26, two kept for the checkout; and on 2026-11-09 the ten runs before took 19 to 23.4 and a merge group was cut at 26.2, so its tests were dealt into legs. Add a leg before one nears the step's limit.
# About 6,400 mutants, in sixteen shards of about 400, each kill confirmed by two runs of one test. With six shards of 168, of about 1,000 mutants then, the legs ran 14 to 25 minutes against a 25-minute timeout, and legs were cut at 25:00 in three pull requests, once 22 ms after the last kill had printed. With eight shards of 651 the legs mostly ran 6 to 14 minutes, and on 2026-11-06 one was cut at its step's 27 minutes on a runner that ran it 2.8 times slower than a re-run of the same tree ("CI: twelve mutant shards, so a slow runner does not cut a leg at its limit", issue 583); the first run of twelve took 4.4 to 12.3 minutes a leg, 7.3 the median. By 2026-11-10, with about 6,390 mutants, the ten merge groups before took 15.8 to 17.3 minutes at the slowest leg's job, and nine kept under 40% of the step's 27 to spare, so the shards became sixteen. Every mutant names the tests that kill it, which run alone ("Mutation run: every mutant names its tests, the slow suites run last, and a builder may break a few", issue 217); a mutant they miss is an ERROR, named_test_does_not_kill, since the whole suite run after them took 10 to 21 minutes of a leg (2026-11-09). Add a shard before a leg nears its timeout, or before a leg of the median time would take more than 22 minutes on a runner three times slower. A runner as slow may still cut the longest leg: run it again.
# A pull request's run is cancelled when a newer push to it starts another: on 2026-11-09, 13 of a day's 47 ran on after a newer push, about 870 runner-minutes on the Free plan's 20 runners. A merge group's run is never cancelled: its group is its own run's id, and it cancels nothing; nor does a re-run, whose attempt is past its first, which would otherwise cancel the run of a newer head. The cancelled run's failed verify is passed over beside a later run on the same head (decision 0203), and decision 0206 amends 0014's row. tools/check_workflows.py holds both lines word for word.
# `verify` checks out the base, never the head, and runs the base's tools/needs_passed.py, so a change cannot make its own verdict pass: decision 0023.
# `commits` checks out the base too and runs the base's tools/commits_are_the_maintainers.py, which reads every commit the pull request carries and refuses unless each names the Maintainer ("Identity: every commit a pull request carries is the Maintainer's, checked outside the shell", issue 204). It reads the event's payload too, and refuses unless an account on the configuration's list opened the pull request from this repo: decision 0102. On a merge group, whose payload names no opener, it reads the pull request the queue's ref names, with this job's read-only token.
name: factory
on:
pull_request:
merge_group:
permissions: {}
concurrency:
group: ${{ format('factory-pr-{0}', github.event_name == 'pull_request' && github.run_attempt == 1 && github.event.pull_request.number || github.run_id) }}
cancel-in-progress: ${{ github.event_name == 'pull_request' && github.run_attempt == 1 }}
jobs:
checks:
runs-on: ubuntu-24.04
Expand Down Expand Up @@ -133,7 +137,7 @@ Its shape, each credential named by one step alone and its upload the one `UPLOA
`tools/check_workflows.py`, by issue 309, "Workflows: eval.yml runs the review Intent's held-out cases and writes eval-report/1".

Org standard: ci-standards.md, "Permissions Policy", "Action Pinning Policy" and "CI Job Naming Convention". Its name, triggers and stages depart from "1. CI Pipeline (ci.yml)": `docs/decisions/0014-where-this-repo-departs-from-the-orgs-standards.md`.
Held by: `tests/test_tools.py::test_a_rule_is_not_dodged_by_writing_the_workflow_another_way`, `tests/test_tools.py::test_an_example_in_the_standard_that_has_drifted_is_caught`, `tests/test_tools.py::test_a_lens_job_that_would_never_report_under_its_name_is_caught`, `tests/test_tools.py::test_the_required_check_fails_unless_every_job_it_needs_passed`, `tests/test_tools.py::test_a_lens_job_that_judges_its_needs_runs_from_the_base`, `tests/test_tools.py::test_the_judging_step_and_its_checkout_are_held_to_their_exact_shape`, `tests/test_tools.py::test_the_mutant_shards_are_told_their_index_and_count_by_the_matrix`, `tests/test_tools.py::test_the_legs_of_make_verify_are_told_their_place_and_count_by_the_matrix`, `tests/test_tools.py::test_the_legs_of_make_verify_run_every_test_once`, `tests/test_evals.py::test_the_eval_jobs_name_is_reserved`, `tests/test_evals.py::test_the_eval_workflow_names_each_credential_in_one_step_alone`
Held by: `tests/test_tools.py::test_a_rule_is_not_dodged_by_writing_the_workflow_another_way`, `tests/test_tools.py::test_an_example_in_the_standard_that_has_drifted_is_caught`, `tests/test_tools.py::test_a_lens_job_that_would_never_report_under_its_name_is_caught`, `tests/test_tools.py::test_the_required_check_fails_unless_every_job_it_needs_passed`, `tests/test_tools.py::test_a_lens_job_that_judges_its_needs_runs_from_the_base`, `tests/test_tools.py::test_the_judging_step_and_its_checkout_are_held_to_their_exact_shape`, `tests/test_tools.py::test_the_mutant_shards_are_told_their_index_and_count_by_the_matrix`, `tests/test_tools.py::test_the_legs_of_make_verify_are_told_their_place_and_count_by_the_matrix`, `tests/test_tools.py::test_the_legs_of_make_verify_run_every_test_once`, `tests/test_tools.py::test_only_a_superseded_pull_request_run_is_cancelled`, `tests/test_evals.py::test_the_eval_jobs_name_is_reserved`, `tests/test_evals.py::test_the_eval_workflow_names_each_credential_in_one_step_alone`

## Pinned to what was looked up

Expand Down
4 changes: 4 additions & 0 deletions examples/workflows/verify.yml
Original file line number Diff line number Diff line change
@@ -1,13 +1,17 @@
# The required check, `factory / verify`: it passes only when `make verify` and every shard of the mutation check pass on the commit, with the history tools/check_decisions.py compares it with. On a pull request the shards decline and pass, and on a merge group each runs in full (record 0182).
# `checks` runs `make verify` in three legs, its simulators on 300 seeds: each leg runs every check and every third test, told its place and the count by the matrix itself, never by a number written in, so that across the legs each test runs once. As one job, on 2026-10-21 the runs that finished took 11 to 14.5 minutes, and six runs in three pull requests were cut at its 15-minute timeout with the tests still running, so its timeout was raised to 25 minutes; by 2026-10-29 a run took up to 23, so the job was given 28 and its step 26, two kept for the checkout; and on 2026-11-09 the ten runs before took 19 to 23.4 and a merge group was cut at 26.2, so its tests were dealt into legs. Add a leg before one nears the step's limit.
# About 6,400 mutants, in sixteen shards of about 400, each kill confirmed by two runs of one test. With six shards of 168, of about 1,000 mutants then, the legs ran 14 to 25 minutes against a 25-minute timeout, and legs were cut at 25:00 in three pull requests, once 22 ms after the last kill had printed. With eight shards of 651 the legs mostly ran 6 to 14 minutes, and on 2026-11-06 one was cut at its step's 27 minutes on a runner that ran it 2.8 times slower than a re-run of the same tree ("CI: twelve mutant shards, so a slow runner does not cut a leg at its limit", issue 583); the first run of twelve took 4.4 to 12.3 minutes a leg, 7.3 the median. By 2026-11-10, with about 6,390 mutants, the ten merge groups before took 15.8 to 17.3 minutes at the slowest leg's job, and nine kept under 40% of the step's 27 to spare, so the shards became sixteen. Every mutant names the tests that kill it, which run alone ("Mutation run: every mutant names its tests, the slow suites run last, and a builder may break a few", issue 217); a mutant they miss is an ERROR, named_test_does_not_kill, since the whole suite run after them took 10 to 21 minutes of a leg (2026-11-09). Add a shard before a leg nears its timeout, or before a leg of the median time would take more than 22 minutes on a runner three times slower. A runner as slow may still cut the longest leg: run it again.
# A pull request's run is cancelled when a newer push to it starts another: on 2026-11-09, 13 of a day's 47 ran on after a newer push, about 870 runner-minutes on the Free plan's 20 runners. A merge group's run is never cancelled: its group is its own run's id, and it cancels nothing; nor does a re-run, whose attempt is past its first, which would otherwise cancel the run of a newer head. The cancelled run's failed verify is passed over beside a later run on the same head (decision 0203), and decision 0206 amends 0014's row. tools/check_workflows.py holds both lines word for word.
# `verify` checks out the base, never the head, and runs the base's tools/needs_passed.py, so a change cannot make its own verdict pass: decision 0023.
# `commits` checks out the base too and runs the base's tools/commits_are_the_maintainers.py, which reads every commit the pull request carries and refuses unless each names the Maintainer ("Identity: every commit a pull request carries is the Maintainer's, checked outside the shell", issue 204). It reads the event's payload too, and refuses unless an account on the configuration's list opened the pull request from this repo: decision 0102. On a merge group, whose payload names no opener, it reads the pull request the queue's ref names, with this job's read-only token.
name: factory
on:
pull_request:
merge_group:
permissions: {}
concurrency:
group: ${{ format('factory-pr-{0}', github.event_name == 'pull_request' && github.run_attempt == 1 && github.event.pull_request.number || github.run_id) }}
cancel-in-progress: ${{ github.event_name == 'pull_request' && github.run_attempt == 1 }}
jobs:
checks:
runs-on: ubuntu-24.04
Expand Down
Loading
Loading