fix(deploy): parent-stability wait is too short for a back-to-back merge-queue deploy #3188

🤖 Generated by the Agentic Engineer

Evidence

platform#3171 was evicted from the merge queue on 2026-08-17 with:

##[error]The Flux image-verification policy owner did not quiesce before the image-verification policy handoff.
kustomization/infrastructure: Ready=True ReconciliationSucceeded: Applied revision: latest@sha256:b6f49e39…
kustomization/infrastructure: Healthy=True Succeeded: Health check passed in 45.411764154s

Run 31993614033, step
🔑 Stage Flux and consumer GHCR pull credential, 04:13:35Z → 04:17:45Z.

This is a distinct cause from the two already known. It is not the autoscaler removal-candidate
race (fixed by #3185) and not the residual GHCR drain fence that evicted the same PR at 00:01Z — that
fence sat on node autoscale-cx43-2834820e83860317, which the autoscaler has since removed, and no
current node carries a ghcr-auth-drain-owner annotation.

Diagnosis

pause_flux_policy_handoff in scripts/refresh-flux-ghcr-auth.sh waits for
flux_policy_parent_is_stable && flux_policy_handoff_is_quiescent. Both reduce to "no
Reconciling=True condition". The budget is SYNC_ATTEMPTS=60 × SYNC_INTERVAL=2s — 120 seconds.

Two things place the blame on the parent-stability half rather than on the handoff target:

1. The failure report prints only the parent (kustomization/infrastructure), and prints
   Ready/Healthy but no Reconciling line — flux_policy_report_conditions emits whichever
   of the three exist, so at the moment of the final snapshot the parent had already settled. That is
   the signature of a budget expiring just before the condition clears, not of a stuck resource.
2. The same message reports the health check alone taking 45.4s, so a large fraction of the
   120s budget is consumed by one phase of a single reconcile.

The trigger is back-to-back deploys, which is exactly what a merge queue produces. #3185's deploy
ran 03:38–04:05Z; #3171's started at 04:13Z and caught infrastructure still reconciling the revision
#3185 had just published.

Confirming the resource is healthy at rest (measured 2026-08-17T04:24Z): interval=60m,
suspend=false, generation == observedGeneration == 1576, and resourceVersion unchanged across
40 seconds of sampling. Nothing is wrong with the Kustomization — the wait is simply shorter than a
real post-publish reconcile.

Audience and impact

Every merge to main. A merge_group failure evicts the PR, so any PR queued shortly after
another one merges can be thrown out for a reason unrelated to its own diff, and re-queuing without
diagnosis just re-hits it.

Expected behaviour

A deploy that starts while the previous revision is still reconciling waits for it rather than failing
the merge group.

Acceptance criteria

• The parent-stability wait tolerates a full post-publish reconcile of infrastructure, sized from
  measured reconcile duration rather than a round number, with the budget derived or documented.
• Exhausting the budget reports the parent's Reconciling condition and elapsed wait explicitly, so
  "settled just after the budget" is distinguishable from "genuinely stuck".
• A regression test in scripts/tests/refresh-flux-ghcr-auth/ covers a parent that becomes stable
  part-way through the wait.
• Re-verified by two merge-queue entries landing back to back without eviction.

Rough size: S. Blocks the merge queue intermittently, so it is worth doing before the next batch of
platform PRs is queued.