Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
15 commits
Select commit Hold shift + click to select a range
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 5 additions & 2 deletions .claude/rules/toolchain.md
Original file line number Diff line number Diff line change
Expand Up @@ -262,8 +262,11 @@ call` with no `CLOUD-*` key **in that same paragraph** stops the lap. Two open
command's exit status: piping it into a pager or filter, following it with `;`
or `||` in the same list, and detaching it with `nohup`/`&`. The
verdict-bearing list is written once as data in the task — `mise run`,
`git push`/`fetch`/`rebase`, mutating `gh pr`, and `cargo` minus its query
subcommands — because the guard was scoped to the literal string `mise run`
`git push`/`fetch`/`rebase`, mutating `gh pr`, `cargo` minus its query
subcommands, and `bats` given a suite (CLOUD-473: `mise run test:bats` was
covered by the `mise` row while `mise exec -- bats` and a path-invoked
`tests/bats/bin/bats` ran the same gates unguarded) — because the guard was
scoped to the literal string `mise run`
and an agent complying with it exactly kept making the identical error on the
next command (CLOUD-199, measured on `git push`, `cargo clippy` and
`cargo test`). `&&` is deliberately allowed: it short-circuits, so a failure
Expand Down
55 changes: 50 additions & 5 deletions .github/workflows/ci.yml
Original file line number Diff line number Diff line change
Expand Up @@ -87,7 +87,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down Expand Up @@ -201,7 +210,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down Expand Up @@ -274,7 +292,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: jdx/mise-action@7e36c90d9ab29c415a2384db3006f3ec8a8cc654 # v4
Expand Down Expand Up @@ -337,7 +364,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
- uses: jdx/mise-action@7e36c90d9ab29c415a2384db3006f3ec8a8cc654 # v4
Expand Down Expand Up @@ -396,7 +432,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/commit-lint.yml
Original file line number Diff line number Diff line change
Expand Up @@ -64,7 +64,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down
16 changes: 16 additions & 0 deletions .github/workflows/fast-forward.yml
Original file line number Diff line number Diff line change
@@ -1,5 +1,21 @@
name: fast-forward

# THE RUN CARRIES THE PR IT ANSWERS FOR (CLOUD-409, CLOUD-456). An
# `issue_comment` run attaches to the default-branch tip, so `head_branch` and
# `head_sha` name `main` for every one of these — there is no field on the run
# object that says which PR asked. `land` therefore had only the timestamp to
# filter on, and at the measured cadence (400 runs in 30 minutes, 243 of them
# refusals) its ~90s window was 243 strangers' refusals, any of which it would
# read as its own verdict. `run-name` surfaces as `display_title` on the run,
# which gives the filter an exact key.
#
# Both interpolations are integers from the event payload. NOT
# `github.event.comment.body`, which is attacker-controlled and is the template
# injection zizmor exists to refuse. The comment id is the stronger of the two on
# its own: it names THIS lap's request on THIS head, so it also subsumes the
# `SINCE` fence against an earlier lap of the same PR.
run-name: fast-forward #${{ github.event.issue.number }} @${{ github.event.comment.id }}

# GitHub's merge button cannot fast-forward: "Rebase and merge" rewrites every
# commit under a new SHA, and "Merge" adds a merge commit. Both discard the exact
# objects CI tested. This workflow lands a PR by *true* fast-forward instead —
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/test.yml
Original file line number Diff line number Diff line change
Expand Up @@ -73,7 +73,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

# `path: action` frees the workspace root for the fixture. The action then
# lives at ./action, and `$GITHUB_ACTION_PATH/Cargo.toml` — which is where
Expand Down
11 changes: 10 additions & 1 deletion .github/workflows/zizmor.yml
Original file line number Diff line number Diff line change
Expand Up @@ -75,7 +75,16 @@ jobs:
run: |
body=$(gh api -H "Accept: application/vnd.github.raw" \
"repos/$GH_REPO/contents/mise-tasks/ci-lease-precondition?ref=main") || exit 0
bash -c "$body"
# `|| exit 0` on BOTH lines, and the second is the one that was missing
# (CLOUD-420). A fetch that fails is handled; a fetch that SUCCEEDS with
# something that is not shell was not — a truncated body, a proxy error
# page returned 200, or a bad edit to the trunk copy makes this a syntax
# error in the FIRST step of every job. The step reds under the runner's
# default `bash -e {0}`, `final` fails its `needs:` assertion under
# `always()`, and `land` re-drafts a healthy PR — fleet-wide, from one bad
# response. The precondition's own header promises it never exits
# non-zero; this is what makes that true at the call site too.
bash -c "$body" || exit 0

- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
with:
Expand Down
104 changes: 104 additions & 0 deletions .serena/memories/workflow/landing-loop.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,104 @@
# The landing loop, and why its shape looks like breakage from an old clone

Read this **before** changing anything under `mise-tasks/land*`, `ci-wait`,
`main-watch`, `checks-green`, or `tests/land*.bats`. Three things here read as
defects to an agent on a clone that predates them, and each is the change:

- **a lease held by a live process**, not a file the tree owns;
- **a raced wait with no wall clock anywhere** — no timeout, no cap, no deadline;
- **verdicts that refuse to answer** rather than guessing (exit 3 is a real answer).

Repairing any of them back is the regression. Four gates now refuse the repair,
listed at the bottom with their bypasses so a refusal can be told from a defect.

## The lap

fetch → rebase → `verify` → `verified` → push → `ci-wait` ∥ `main-watch` →
`/fast-forward` → read the answer → lap.

A refusal is **the design working**, not a failure. Each lap rebases onto a
little more landed work, so conflicts arrive one small resolvable increment at a
time; batching laps removes no refusal and only makes each one bigger. An agent
that infers "a race I keep losing" and wraps the loop in bespoke retry is
optimising against the design — measured, CLOUD-238.

`LAND_MAX_LAPS` is a **count** of laps, `LAND_LOCK_MAX_WAITS` a count of lease
turns lost, `LAND_ANSWER_MAX_UNKNOWNS` a count of unreadable bot answers. None is
a clock. A hang is fixed by an exit condition that can fire; a wall-clock cap
reintroduces the VM-reap gap it was meant to close and lands as a false
"refused" on a slow bot.

## The lease

`land-lock` is a compare-and-swap on `refs/heads/batten-land-lock` — one
operation, no service, no API. Four things the design was pressure-tested into,
each of which cost an incident:

- **It is a BRANCH.** The agent proxy 403s a push outside `refs/heads`, and
GitHub does not enforce the fast-forward rule off `refs/heads` either — a
parentless orphan `PATCH` with `force:false` was _accepted_ on a custom
namespace. The atomicity the design rests on exists only on `refs/heads`.
- **Renewal is `--force-with-lease=<ref>:<observed>`**, a true CAS. `PATCH` with
`force:false` does not give one. Create stays a plain push, so acquire is an
atomic test-and-set.
- **The body carries a nonce.** Git addresses by content, so two mints agreeing
on holder and expiry produce the same sha, and pushing a sha the ref already
holds is an "up to date" no-op reporting success — a rejected claim reading as
a win.
- **Rolling TTL (120s) with a 30s heartbeat**, not a static `p95 x 3`. A static
TTL must bound how long a hold might legitimately take, which is a guess wrong
in both directions; a rolling one bounds only how long until we notice a holder
stopped beating. Three missed beats is the Raft/etcd margin.

Enforced at three points, deliberately: `ready-guard` (free, local, a receipt),
`ci-lease-precondition` (server-side, costs a cancelled matrix), and `land`
itself. The first is a fail-fast convenience — a hook can be unloaded — so the
CI-side check is the load-bearing half. Every one of them **fails open**: a gate
that cannot look must never become a gate that blocks everything, because the
cost of failing open is one matrix and the cost of failing closed is the fleet.

## Why there is no clock, anywhere

`ci-wait` is unbounded by design and `tests/land.bats` asserts the task carries
no wall-clock timeout. The tests are held to the same bar, and the shape to copy
when a case genuinely needs a precondition the runner might not create is
CLOUD-448's, in `tests/land-lock.bats`: **retry the SETUP, never the
MEASUREMENT** (retrying the measurement is drive-to-green), and `skip` with a
diagnostic naming the ticket when N attempts fail to establish it.

The failure this prevents is subtler than a flake: a guessed `sleep` standing in
for "the background thing has happened" makes a case pass **vacuously** on a
loaded box — the process is alive because it never ran, so a regression is
invisible on exactly the runs where the suite is slowest.

## Verdicts

- One roster, `CI_REQUIRED_CHECKS` in `mise.toml [env]`, read by `checks-green`
and by `land`'s `graded_runs`. Two copies drifted once and that was CLOUD-327.
- **Each name is judged by its LATEST run** — a sha accumulates a check-run per
event, and a draft-created head carries its `opened` skip set forever.
- `skipped` and `cancelled` are **not** bad conclusions; they are the absence of
an answer, and reading either as red wedges a branch with no exit. `absent` is
different again and is tolerated only where a workflow is path-filtered.
- Exit **3** is "no answer yet" and is a first-class outcome, not a failure to
decide. Exit **2** is "could not look", and it must never be reachable from a
path that would otherwise spend a matrix.
- The fast-forward verdict is keyed to the PR's own comment id via the
workflow's `run-name`, because an `issue_comment` run attaches to the
default-branch tip and carries nothing else that identifies the PR. Reading it
by timestamp alone means reading strangers' refusals as your own.

## The gates that refuse a repair

| Gate | Refuses | Bypass |
| ----------------- | ---------------------------------------------------------------------------------------------- | -------------------------------- |
| `ci-local-parity` | a `pull_request` job with no lease precondition; a precondition body run without `\|\| exit 0` | — (gate, not a hook) |
| `ready-guard` | `gh pr ready` without verify + linear-check receipts for this exact HEAD, or without the lease | `BATTEN_READY_GUARD_BYPASS=1` |
| `run-shape-guard` | discarding a verdict-bearing command's exit status | `BATTEN_RUN_SHAPE_BYPASS=1` |
| `contract-drift` | nothing — it reports, once, that the surface moved under a running session | `BATTEN_CONTRACT_DRIFT_BYPASS=1` |

## Rollout posture

Every mechanism here fails open on a clone that predates it, so none of them
binds an agent that never fetches. `branch-age-check` is the backstop: a branch
old enough to be dangerous is already failing a gate for being old.
8 changes: 4 additions & 4 deletions AGENTS.md
Original file line number Diff line number Diff line change
Expand Up @@ -34,6 +34,7 @@ auto-loaded. Read the matching one at its trigger; don't reconstruct the detail.
| `core` | navigating `crates/batten/src`; asking what a module does or where to add one |
| `workflow/board-states` | starting/finishing a `CLOUD-*` issue; reasoning about what is in flight |
| `workflow/agent-fanout` | spawning a subagent, or running more than one session against this repo |
| `workflow/landing-loop` | landing a branch; before "repairing" `land`, the lease, the CI wait or their suites |
| `github-access` | any GitHub op; before claiming the toolchain/CI "can't reach GitHub" |
| `github-rest-etiquette` | writing a task that calls the GitHub API; diagnosing a 403/429/abuse response |
| `toolchain-and-hooks` | pinning a tool, adding a task, touching `hk.pkl` or the gate |
Expand Down Expand Up @@ -126,10 +127,9 @@ run in the web sandbox — read `mem:github-access` before doubting.)
branch is rebased on current `origin/main`. "Green but stale" is not green.
3. **`mise run linear-check`.** Don't ready by hand: `land` readies, after its
push, and a ready spent before that buys only draft-era skips (CLOUD-247).
4. **`mise run land`, backgrounded.** It drives the whole loop — rebase →
`verify` → push → wait → `/fast-forward` → lap — racing `ci-wait` (**no timeout,
no cap, never the PR webhook**) against `main-watch`. It stops for three things
only: a rebase conflict, a failed `verify`, or red CI, re-drafting the PR.
4. **`mise run land`, backgrounded.** It drives the whole loop — **no timeout, no
cap, never the PR webhook** — and stops for three things only: a rebase
conflict, a failed `verify`, or red CI, re-drafting the PR. `mem:workflow/landing-loop`.
5. **Never re-run CI on an already-tested SHA.** Fast-forward means `main` takes
the PR's exact, already-passed commits. Don't add push-to-`main` triggers.

Expand Down
2 changes: 1 addition & 1 deletion fuzz/Cargo.lock

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

Loading