Repository navigation
fix(policy): image-verification webhook timeouts (context deadline exceeded) can wedge the merge queue #2431
Description
🤖 Generated by the Daily AI Assistant
Problem
The verify-app-images Kyverno ImageValidatingPolicy webhook (ivpol.{mutate,validate}.kyverno.svc-fail, timeout=10s, fail-closed) intermittently returns context deadline exceeded while doing keyless cosign verification (which reaches Fulcio/Rekor/ghcr at admission time). When that happens during a deploy-prod merge_group run, legitimate first-party app pods are denied inside the 20-minute apps health-check window → the apps Kustomization times out → the whole merge_group is evicted.
Observed live on the wedding-app rollout: RS wedding-app-684f95779 logged FailedCreate … ivpol.mutate.kyverno.svc-fail … context deadline exceeded ×19 over ~4h, interleaved with clean policy responses. This transient timeout is independent of policy correctness — even a correctly-trusted image can be denied when verification is slow, and in the merge queue that stalls every platform PR, not just the app being rolled out.
Why it matters
A network/latency blip in sigstore or the registry shouldn't be able to wedge the entire platform merge queue or stall an app rollout. Today it can, because image verification is a fail-closed 10s webhook on the hot admission path.
Proposed direction (investigate, pick one)
- Raise the image-verification webhook timeout (10s → ~30s) so cold cosign verifications complete.
- Enable Kyverno's cosign verification-result caching (and/or warm the TUF/Rekor cache) so repeat admissions don't re-hit the network.
- Consider audit-with-retry for first-party images so a transient verify failure doesn't hard-deny a legitimate, already-published image.
Rough size
S–M: a Kyverno HelmRelease values / policy webhook-config change plus a short investigation of which mitigation is safe (must not weaken the security posture the policy exists for).
Activity
fix(kyverno): raise webhook timeout to 30s so cold cosign verifies do…
devantler commentedon Jul 4, 2026
Corrected root cause (verified live on prod) —
--webhookTimeoutdoes NOT fix this.I tried the obvious global lever (
admissionController.container.extraArgs.webhookTimeout: 30) in #2432 and deployed it. It applied to the cluster, but checking the resulting webhook configs proves it doesn't touch the image-verification path:timeoutSecondsafter--webhookTimeout=30validate.kyverno.svc-fail(legacy ClusterPolicy engine)ivpol.{validate,mutate}.kyverno.svc-fail(ImageValidatingPolicy engine)So the flag only raises the legacy policy webhooks. The new
policies.kyverno.ioIVPOL webhooks — which run our cosign image verification — keep the hardcoded 10s, and wedding-app stayed blocked with…/verify-app-images/verify-ksail-images?timeout=10s: context deadline exceeded. #2432 is reverted to draft.The IVPOL webhook timeout has only two real levers in Kyverno v1.18.1, both bigger than a values tweak — this needs a call on which way to go:
spec.webhookConfiguration.timeoutSeconds: 30on it. The finegrained-webhook trap documented inverify-ksail-images.yamlonly breaks with multiple IVPOLs (concatenated URL path → "policy not evaluated"), so a single policy should register a clean finegrained webhook. Trade-off: merges two security-critical policies (app + ksail identities) into one CEL set; the "single IVPOL registers cleanly" assumption must be verified on the cluster before trusting it (getting it wrong fail-closes all Pod admission).webhookConfigurationon both existing IVPOLs. Trade-off: a controller version bump on the prod admission path — needs its own validation.Both are prod-admission-critical changes I don't want to ship blind from an unattended run. @devantler — which direction do you prefer? I can prepare and validate whichever you pick. Meanwhile the old wedding-app pod is still serving (no user-facing outage); the stuck items are the v1.14.1 rollout and the merge queue (every
deploy-prodhealth-checks wedding-app and evicts).devantler commentedon Jul 4, 2026
Root cause of the CURRENT wedge corrected again — it is the #2429 dual-identity attestor, not the webhook timeout. Live kyverno admission logs show the wedding-app verify completes in ~3s and fails with
unsupported: multiple identities are not supported at this time— cosign v3's bundle path (sigstore-go) rejects any attestor with >1 identity entry, so #2429's transition attestor fail-closes every image it gates. Fix: one identity with an alternatedsubjectRegExp→ #2456 (verified:cosign verifyof the blocked image against the alternated regexp exits 0). Same bug shape as #2399→#2404 at the Flux OCI layer.The intermittent
context deadline exceededonivpol.mutatethis issue tracks is real but secondary (self-heals via controller retries; last seen under deploy load). Once #2456 lands, this issue drops from "queue-wedging" to "hardening": the two levers from my earlier comment (merge the IVPOLs + per-policywebhookConfiguration, or a Kyverno bump once a release ships a finegrained fix) remain the options, at normal priority. Note: Kyverno v1.18.1 is still the latest release, so the bump lever isn't available yet.devantler commentedon Jul 28, 2026
This recurred in merge-group run 30338959344: the fail-closed
verify-ksail-imagesmutating webhook timed out three consecutive runtime-probe admissions after the GHCR synchronization Lease had been acquired and renewed successfully.Current live evidence:
PR #2836 now sets both first-party IVPOLs to the fail-closed 30-second maximum and raises the pre-reconcile same-name probe budget from three to six attempts. Persistent denial still blocks every mutation. RED/GREEN contracts, both KSail overlay validations, and server-side policy dry-runs pass at head
216407bc42312b4ea55f558be250a4023f8fb245.devantler commentedon Jul 28, 2026
Follow-up from merge-group run 30340663166: retrying six times was insufficient because this deploy proves runtime pulls before publishing the candidate policy manifests. The live Kyverno webhook therefore remained the combined 10-second path and exhausted all attempts on one node.
PR #2836 now closes that bootstrap edge under the existing GHCR synchronization Lease:
failurePolicy: FailIVPOLs/ivpol/mutate/verify-app-imagesand/ivpol/mutate/verify-ksail-imageswebhooks at 30 secondsThe new pre-probe ordering and non-convergence refusal cases were RED before implementation and are GREEN in the 181-test transaction suite. Syntax, ShellCheck, the dedicated safety harness, and both 115-kustomization/555-file KSail validations pass.
devantler commentedon Jul 28, 2026
Correction from merge-group run 30344061593: separate fine-grained paths in Kyverno v1.18.2 fix the old concatenated-URL defect, but they do not make multiple timeout-bearing IVPOLs safe.
The app runtime probe was denied with:
Policy verify-app-images failed: policy not evaluatedKyverno's v1.18.2 image-verification engine constructs the mutation outcomes annotation from only the policies selected for the current webhook and writes it with an
addoperation. With two fine-grained mutating webhooks, the later policy replaces the earlier policy's outcome; the corresponding validating webhook then sees its policy missing and denies. This is structural, not an informer-convergence delay, so waiting or retrying cannot repair it.PR #2836 now implements the issue's original single-IVPOL direction:
failurePolicy: Failpath before any runtime probe or drainCandidate rejection, covered-policy deletion failure, legacy/combined paths, fail-open webhooks, and non-convergence are all negative-tested. The full transaction suite, safety harness, ShellCheck, and both 115-kustomization/554-manifest overlay validations pass at signed head
4552eee0.