Repository navigation
A PR whose deploy succeeds is still evicted, because the heal path holds prod-deploy past the queue timeout #3091
Description
🤖 Generated by the Agentic Engineer
One failed prod deploy deterministically costs the next queued PR its slot, even when that next
PR's own deploy succeeds completely. This is a separate mechanism from #3089: there the deploy
failed, here it passed and the PR was evicted anyway.
Evidence
Measured on the 2026-08-11 14:00–15:40Z queue window.
#3085 failed its evidence gate and was evicted — that part is #3089. What matters here is what its
failure did to the PR behind it:
| When | What |
|---|---|
| 14:19:44Z | #3084 added to merge queue |
| 14:20:29Z | #3085 deploy fails (evidence gate) |
| 14:20:35Z | #3085 🩹 Heal Prod starts, taking the prod-deploy lock |
| 14:21:04Z | #3084 merge group builds |
| 14:59:11Z | #3085 heal finishes — 38m36s holding the lock |
| 14:59:16Z | #3084 deploy finally starts, 5s after the lock frees |
| 15:23:45Z | #3084 evicted by github-merge-queue[bot] — 64 min after enqueue |
| 15:40:32Z | #3084 deploy succeeds — 17 min after it was already evicted |
Run 31500671569 (#3085) and
31501050429 (#3084).
#3084 merge_group conclusion: success. PR state: still open, never merged.
Why it happens
Three settings interact, each individually reasonable:
- The
Require merge queueruleset setscheck_response_timeout_minutes: 60. deploy-prodandheal-prod-on-failureshareconcurrency: group: prod-deploywith
cancel-in-progress: falseandqueue: max— deliberately, so a deploy is never dropped or run
concurrently against prod.- A prod deploy takes 36–64 min, and the heal path takes a comparable ~38 min.
So after any failed deploy the next entry's wall-clock is heal (~38m) + deploy (~41m) ≈ 79m, which
exceeds the 60-minute timeout before its own deploy can finish. The queue gives up on a PR whose
checks were going to pass. Because the evicted PR must then be re-enqueued and re-run a full deploy,
one flaky deploy can keep displacing the queue for hours.
max_entries_to_build: 1 does not help: the entry is built promptly, it is the job that is stalled
behind the lock, and the queue timer runs from enqueue regardless.
Impact
- A correct, fully green PR is silently dropped from the queue and has to be noticed and re-enqueued
by hand. - Each occurrence burns a complete prod deploy cycle (~41 min of deploy against the real cluster)
that is thrown away — the deploy ran, succeeded, and merged nothing. - Reading the run afterwards is actively misleading: the merge_group run says
success, so the
eviction looks unexplained unless the timeline is checked.
Expected behaviour
A PR whose merge-group checks all succeed is merged, not evicted, even when it waited on the
prod-deploy lock behind an unrelated heal or deploy.
Acceptance criteria
- The merge queue's
check_response_timeout_minutesexceeds the realistic worst case of
heal + deploy(measured ~79 min; suggest 150 for headroom), or the deploy is taken off
the critical path so the queue timer is not gated on a lock the entry does not hold. - A merge_group run that finishes successfully after its PR was already evicted is surfaced
rather than silently discarded — the eviction reason is visible without reconstructing the
timeline by hand. - A failed deploy's heal path no longer starves the following entry for its whole duration, or
the queue accounts for that wait.
Note the first criterion is a repository ruleset change, not a code change, so it needs the
maintainer to apply it.
Related: #3089 (the transient that caused this window's first failure), #3065, #2957.
Activity
devantler commentedon Aug 11, 2026
Reproduced live ~90 minutes after the window above, and the second run surfaced a worse consequence
than the delivery cost this issue was filed for.
Second occurrence
#3086enqueued (fully green, Codex review clean at head)#3084enqueued#3090's deploy fails#3090evicted#3086's merge group builds — 5s later#3090's🩹 Heal Prodstarts, takingprod-deployfor ~38 minSame shape as the first: the heal from a failed deploy starts moments before the next group builds,
so the next entry's deploy cannot start until ~16:47, needs ~41 min, and finishes ~17:28 — past its
~17:09 timeout. Eviction was arithmetically certain before the deploy had run a single step.
I dequeued
#3086and#3084rather than let them proceed, because of the following.The more serious consequence: a successful deploy that is evicted leaves prod ahead of main
heal-prod-on-failureruns only when the deploy fails. Its own comment states the reason itexists:
The timeout eviction produces the same end state through the opposite path. In the first window
#3084's deploy succeeded at 15:40:32Z, so heal wasskipped— correctly, by its owncondition — but the PR had already been evicted at 15:23:45Z and never merged. So prod was deployed
from a speculative ref that is not on
main, and nothing restored it.That is the case the heal path does not cover: it guards
deploy failed → ejected, and this isdeploy succeeded → ejected anyway. The drift is self-correcting on the next successful deploy frommain, but until then prod runs code that was never merged, and no signal is emitted.Adjacent to but distinct from #2828, which is about heal overwriting a newer successful deploy.
Additional acceptance criterion
prod to
mainthe same way a failed deploy does — the heal condition keys on "this group wasejected", not on "the deploy failed".
devantler commentedon Sep 6, 2026
The existing ejected-group recovery criterion also reproduced for manual dequeue on 2026-09-06. PR #3627 was temporarily removed to complete a review-contract step; this exposed an obsolete deployment starting after removal.
devantler; complete GraphQL timeline and null queue entry confirmed removal.Exact binding: merge-group commit
4ace8380a2bc5b2718e66124baf140e19c0648cc, PR headfe4eba0702d3b61e8663d6eabf387a1738860e1f, base19642fe908e2503e2792af0dcc2bfb3a21c2c0e1. The merge-group tree matches the inspected source. At that revision,deploy-prodchecks the original event and changed paths, while the production composite has no live queue-membership guard. Recovery handles failure/cancellation and explicitly excludes successful deployment.This occurrence proves an obsolete deployment start, not successful publication of an unmerged artifact: cancellation intervened, and no observed production revision established that the speculative artifact reached the cluster. After recovery, the source and both infrastructure Kustomizations were Ready and agreed on the restored artifact revision.
Suggested refinements to the existing acceptance criterion:
These extend the existing acceptance comment. The intentional pre-merge production validation remains the delivery model. This is separate from #3097's timeout-specific trigger, and no duplicate issue is needed.
devantler commentedon Sep 6, 2026
The current CI workflow exposes a prerequisite gap relevant to this issue:
deploy-prod.needsomitsvalidate-matcher-efficacy, althoughci-required-checksincludes both jobs. Recovery is conditioned on deployment failure or cancellation.A required sibling check can therefore fail without preventing deployment from beginning. Include this case in the recovery regression coverage and require applicable validation prerequisites before deployment. Preserve the shared production lock and protection for newer legitimate or already-landed revisions. The successful-but-ejected acceptance criterion remains open.
chore: agent-claim #3091 nonce=4e753062ff9f9ec3db9631b718dba81c
fix(ci): restore main when a successful merge-group deploy was alread…
devantler commentedon Sep 27, 2026
Re-checked against
mainon 2026-09-27, before starting work. Two of the three criteria are now met by later changes. The one left is a ruleset setting.🧾 Check the deployed PR is still queuedjob (merge-group-queue-membership) runs after a successful deploy. When the PR is no longer queued,heal-prod-on-failurenow runs on that success path too, so prod is not left ahead ofmain.deploy-proddid not wait forvalidate-matcher-efficacydeploy-prod.needsnow lists every merge-group validator thatci-required-checksenforces, andscripts/guard-deploy-waits-for-merge-group-validators.shkeeps it that way (#4148).mainruleset still hascheck_response_timeout_minutes: 60, while heal plus deploy measured about 79 minutes.The last criterion needs a change to a repository ruleset. Agents cannot make that change, so it is the maintainer's to apply: raise
check_response_timeout_minuteson the platformmainmerge-queue ruleset to about 150. The other way to meet it is to stop a heal from holding the lock the next entry waits on, which is what platform#4192 (held heal) moves toward. The timeout itself is tracked on its own in #3097, which has the later measurements (including #3922 on 2026-09-20).Blocker: authority — the merge-queue ruleset timeout (
check_response_timeout_minutes) can only be changed by the maintainer.devantler commentedon Sep 27, 2026
Correction: the merge-queue timeout is managed as code in devantler-tech/.github (
deploy/repository-rulesets/require-merge-queue-on-platform.yaml, Observe-only today), so it is not maintainer-only. The safe promotion path, and the bypass-actor evidence it needs first, are recorded on #3097, which owns the timeout.devantler commentedon Sep 29, 2026
Status update, 2026-09-29. The authority blocker recorded above no longer holds.
mainmerge-queue ruleset is now managed as code from devantler-tech/.github (released as v2.5.0 through Merge queue evicts PRs after 60 min while merge_group deploys take 36-80 min, so successful deploys merge nothing #3097's work). Its livecheck_response_timeout_minutesis 90, applied at 21:30:53Z today, and the ruleset has not been rewritten since.This issue stays open until #3097 records that evidence. If a merge group is still evicted at 90 minutes, raise the value again in the same ruleset file in devantler-tech/.github. No maintainer action is needed.
devantler commentedon Oct 2, 2026
Closing: all three acceptance criteria are met
Re-checked on 2026-10-02 against the live ruleset and the 2026-09-27 code review above.
Require merge queueruleset (active, last updated 2026-09-29 21:30Z) hascheck_response_timeout_minutes: 90, above the measured worst case of about 79 minutes.Whether 90 minutes holds against a real deploy that runs past 60 minutes is an observation, and #3097 already owns it as its open second criterion. If that observation shows an eviction, it is fixed there by raising the value in the ruleset file in devantler-tech/.github. Keeping this issue open as well would only park the same wait twice.