Repository navigation
Kyverno policy webhook timeouts evict healthy changes from the merge queue #5692
Description
🤖 Generated by the Agentic Engineer
Evidence
On 2026-11-01 a production deploy in the merge queue failed at Flux reconciliation, and #5681 was removed from the queue at 17:46 UTC (run 41331622603). The failure had nothing visibly to do with that pull request:
ClusterPolicy/restrict-arc-retirement dry-run failed: failed calling webhook "validate-policy.kyverno.svc" ... context deadline exceeded
- The deploy tried 3 times over 85 seconds, then stopped. It treats this as a permanent failure.
- Flux itself reconciled the same layer successfully about two minutes later, with no one acting.
- The same timeout was seen after deploys at 02:21 and 06:40 UTC the same day. Both cleared on their own.
A likely cause, not yet proven. That webhook allows 10 seconds. In the log of one Kyverno admission replica, client-side throttling lines ran from 17:43:31 to 17:43:51 while it walked the cluster's API groups. A policy update in that same window ended in a write timeout. In Kyverno 1.19.2, policy validation reads API discovery inside the webhook call, and the cache behind it is invalidated on a timer. The client library logs a throttled request only after a wait over 1 second, and at most once a second, so these lines are a sample: the queue stayed over a second long for about 20 seconds at Kyverno's default limit of 100 requests a second.
| Minute (UTC) | Throttling lines on that replica | Reconcile |
|---|---|---|
| 16:36 | 6 | No failure seen |
| 17:00 | 4 | No failure seen |
| 17:43 | 28 | Failed |
The other replica logged none, but its retained log covers only about the last 30 minutes, including 17:43. The cluster serves 124 API versions, no custom resource definition was added in the last five days, and CPU on the admission replicas was low when read afterwards. The Helm release sets no Kyverno client rate limit.
Not proven: that the timed-out call was the one waiting in that queue, or what produced so many requests. Logs from the two earlier timeouts are gone.
Problem and who it affects
Every queued platform change waits behind the queue head, and an eviction costs the whole queue a deploy cycle of 15 minutes or more, plus a repair deploy. The evicted pull request also looks at fault when it is not.
Options
- Remove the slow walk: find what starts it, then raise the rate limit or otherwise keep it from blocking the webhook. This fixes the cause, if the cause holds.
- Let the deploy wait it out: treat this webhook timeout as retryable for a few minutes. It is cheap and needs no proof of cause, but it hides the slowness and lengthens a bad deploy.
Start with 1, and use 2 only if the cause cannot be shown.
Acceptance criteria
- The cause is shown or refuted by capturing a timed-out call next to the admission replica's log from the same moment.
- A policy dry-run stays under the webhook's limit during a discovery walk, or the deploy survives a timeout that clears on its own.
- A real failure, such as an invalid policy or a webhook that stays down, still fails the deploy.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
chore: agent-claim #5692 nonce=81f8fdf55b37b3b61c63174c026fc745
chore(security): approve the Kyverno rate limit in the authorization …
devantler commentedon Nov 1, 2026
Another sample of this failure, before the fix has deployed. Merge-queue run 41346446390 stopped at 20:36 UTC on 1 November: all 3 infrastructure reconcile attempts failed on the ClusterPolicy dry-run, with the same
validate-policywebhook deadline (10 seconds). It evicted #5687, which changes one allow-list file, and Heal Prod succeeded. #5693 is third in the queue, so this still says nothing about whether the fix works.devantler commentedon Nov 1, 2026
More evidence for the throttling explanation, from the failure that removed #5681 from the queue at about 21:06 UTC on 1 November.
ClusterPolicy/add-security-context dry-run failedon thevalidate-policywebhook (10-second limit) at 21:03:59.So this is now four evictions or stopped deploys today with the same message (15:28, 17:46, 20:36, 21:04). #5681 and #5687 are out of the queue. Per the merge policy they should go back after #5693 has deployed, not before: re-queuing now would likely repeat the failure.
What would show the fix works, once #5693 has deployed: the
validate-policydeadline stops appearing on later queue deploys, and those throttling lines on discovery reads get shorter or stop.devantler commentedon Nov 1, 2026
A second, separate contributor to these evictions is filed as devantler-tech/ksail#8115: KSail's three reconcile attempts appear to re-read one failed result, so a single webhook timeout fails the deploy. #5681's second eviction at 21:06 UTC (run 41347954355) has the same signature as the sample above. #5681 itself is not at fault; it is unqueued and CLEAN at f425817.
devantler commentedon Nov 2, 2026
First reading after #5693 deployed. The issue closed on merge, so this is the check the fix's PR said was still owed.
--clientRateLimitQPS=300and--clientRateLimitBurst=600, generation 14 observed, 2 of 2 ready, no restarts.context deadline exceededline and novalidate-policywebhook failure.This does not yet prove the fix. The failure came and went on its own before: 4 hits on 1 November, with successful deploys in between. One clean deploy cannot separate the fix from that. The throttling reading is the stronger signal, because it was present at each failure read before, though the logs never tied it directly to the webhook call. I will read the next queue deploys for the same message and reopen this issue if it returns.
devantler commentedon Nov 2, 2026
Three queue deploys now read since #5693: runs 41367259048, 41369506535 and 41371279947 have no
context deadline exceededline, and the second one's eviction was a different failure, recorded on #5170. That is still a short sample against a failure that came and went on its own, so I am not calling it proven. The issue stays closed unless the message returns.