Repository navigation
drain-autoscale-node-storage retrofit never fires: CREATE-only trigger starves mutate-existing UR generation #2392
Description
🤖 Generated by the Daily AI Assistant
Problem. The drain-autoscale-node-storage ClusterPolicy (the mutate-existing retrofit half split out in #2378) has never produced a single UpdateRequest. Live evidence, 2026-07-02 ~22:10Z (~28h after admission, policy Ready since 17:56Z):
- All 4 pre-existing
autoscale-cx33-*Longhorn Node CRs still havespec.allowScheduling: true(the policy should setfalse+evictionRequested: true). kubectl get updaterequests -A→ 0.- 24h of kyverno-background-controller logs contain zero lines for this policy, while
propagate-reloader-to-flagger-primary(also mutate-existing) logs "successfully created UR on policy update" for its targets every hour on the controller resync.
Root cause. The rule's match — which doubles as the trigger enumeration for mutateExistingOnPolicyUpdate and the hourly background resync — carries operations: [CREATE]. Existing resources enumerated in background context never satisfy a CREATE-only filter, so the retrofit selects nothing, ever. The in-file comment claiming the hourly background scan runs "regardless of triggers" is empirically wrong (see log comparison above). The filterless propagate-reloader-to-flagger-primary match is the proven working pattern.
Impact. The 2026-06-30 FinOps goal is silently unmet: Longhorn replicas stay schedulable on the 4 autoscale nodes, which keeps them CA-unremovable (cluster stuck at 10 nodes; see #2360 aftermath) — ongoing cost.
Proposed direction. Drop the operations: filter from the match (mirror propagate-reloader). Node-CR churn then re-enqueues URs, but the patch is an idempotent 2-field re-assert — same trade the reloader policy already makes cluster-wide. Rough size: XS (one manifest + comment).
Activity
botantler-1 commentedon Jul 2, 2026
🎉 This issue has been resolved in version 1.94.2 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀
devantler commentedon Jul 2, 2026
Reopening — #2393's fix deployed cleanly but the retrofit STILL does not fire; diagnosis corrected
Live evidence (2026-07-02 ~19:52–21:00Z):
policy_controller.go:259 "updating policy" name=drain-autoscale-node-storageat 19:52:50Z), policy generation 2, Ready.operationsfield at all — yet zero UpdateRequests were created in the following hour and the 4autoscale-*Longhorn Node CRs still haveallowScheduling: true.propagate-reloader-to-flagger-primarylogsmutate.go:56 "successfully created UR on policy update"right after its own "updating policy" line — drain-autoscale never does.nodes.longhorn.ioverified: list/get/watch/update/patch all yes.Code trace (kyverno v1.18.1): the drop is silent by construction at v=2. Path:
updatePolicy→syncPolicy→handleMutate(pkg/policy/mutate.go:13) →getTriggers→ per-triggerhandleUpdateRequest(pkg/policy/policy_controller.go:519) →ApplyBackgroundChecks:len(engineResponse.PolicyResponse.Rules) == 0→return true, nil(silent skip, no log at any level — policy_controller.go:531-533);So the engine evaluation rejects the rule per trigger (or produces no rule response), and at our log level nothing is emitted.
getTriggers' own failure modes (list error) WOULD log an error; none appears — so triggers are likely enumerated and the engine match/evaluation is what rejects.Discriminators still open (both policies have
skipBackgroundRequests: true, both mutate-existing with targets — so it is NOT that field alone): group-qualified custom kindlonghorn.io/v1beta2/Nodevs plainDeploymentin the engine'sCheckKind; thenames: [autoscale-*]block;patchStrategicMergevs the reloader's mutation type; spec-level vs rule-levelmutateExistingOnPolicyUpdate.Proposed next steps (in order):
V(4) "skip creating URs on policy update" rule.Statusline and name the failing check directly. Cheap and definitive; revert after capture.names, switch kind form, rule-level flag).handleUpdateRequest's zero-rules skip deserves at least a V(2) log with the engine response — candidate kyverno issue once root-caused.Interim unwedge still stands (maintainer one-click):
(Exactly what the merged #2378 intends; Longhorn rebuilds replicas before removal; CA can then consolidate the 4 stranded autoscale nodes.)
devantler commentedon Jul 2, 2026
Diagnostic shipped as draft #2396: raises the background controller only to
--v=4viabackgroundController.extraArgs(renders after the features-derived--v=2; klog last-wins — verified on a local chart-3.8.1 render). Once deployed, the next policy re-apply / hourly background scan will log whichApplyBackgroundChecksbranch rejectsdrain-autoscale-node-storage, letting us bisect the 4 discriminators vspropagate-reloader. Revert follows once captured.devantler commentedon Jul 2, 2026
Root cause found — the
-v=4logs (#2396) caught it within seconds of the controller restart. The policy controller DOES enumerate the 4 autoscale Node CRs and callscreating new UR for mutate, but every one is rejected atengine/internal/match.go:29:configuration resource filters doesn't match resource(new.kind=Node new.name=autoscale-cx33-… new.namespace=longhorn-system).The gate is the chart-default
resourceFiltersentry[Node,*,*]— resource filters match on the kind string alone (no API group), so the core-Node exclusion also swallowsnodes.longhorn.ioNode CRs. That's the true discriminator vs propagate-reloader (its kinds aren't filtered); theoperationsfilter (#2393) was a co-factor, not the cause. The admission half (restrict-storage-to-baseline-workers) is equally gated for new autoscale nodes.Fix: #2397 (subtractive
config.resourceFiltersExcludeon the kyverno HelmRelease). The #2396 verbosity revert follows once URs are confirmed live.botantler-1 commentedon Jul 3, 2026
🎉 This issue has been resolved in version 1.94.3 🎉
The release is available on GitHub release
Your semantic-release bot 📦🚀
devantler commentedon Jul 3, 2026
Post-merge confirmation: #2397's fix worked end-to-end. Verified live 2026-07-03 02:10Z — all 4 autoscale-cx33 Longhorn Node CRs flipped to
allowScheduling=false+evictionRequested=true, and 0 Longhorn replicas remain on autoscale nodes (evacuation complete). The temporary v=4 diagnostic (#2396) is being reverted in #2400.