Repository navigation
fix(kubescape): kubevuln produces no CVE data — relevancy ScanCP aborts on partial ApplicationProfiles #2450
Description
🤖 Generated by the Daily AI Assistant
Part of #2447.
Problem
kubevuln is alive and processing its queue (grype DB loads fine at startup; thousands of scan started), but no image carries any vulnerability data. Verified against live prod (2026-07-04):
vulnerabilitymanifestsummaries: 109/109 show.all=0 and.relevant=0, withvulnerabilitiesRef.alland.relevantboth empty (kind:"",name:"").vulnerabilitymanifests: 197/197 have zeropayload.matchesand notool.name— SBOM shells with no grype match phase ever recorded.- Dominant kubevuln log (~815×/24h):
service warning - ScanCP … container profile is partial (workload restart required).
ScanCP is the scan-with-container-profile (relevancy) path — and it is the only path wired. It aborts before grype runs because the ApplicationProfile never reaches completed (123/288 profiles are still partial), so neither .all nor .relevant is ever written. Real registry errors (ScanCVE i/o timeout on postgres:latest) are rare (3×/24h) — this is not a grype-DB or registry-auth failure.
Consequence: the cluster is not genuinely 0-CVE, and #2149 ("reduce reachable CVEs flagged by Kubescape") currently has nothing to act on — nothing is being flagged.
Proposed direction (CVE = both — settled with the maintainer)
- Get grype output flowing now — decouple/enable a full non-relevancy scan (the
.allview) so grype produces CVE matches regardless of ApplicationProfile completion state. Confirm the kubevuln/node-agent config that governs the relevancy-vs-full scan path (relevancy on but not gating the.allwrite). - Complete the profiles for the reachable view — a controlled, batched rollout of the 123 partial-profile workloads so
ScanCPcompletes and the.relevantview (the ~90%-noise-reduction reachable-CVE list) populates. Batch it; do not mass-restart prod at once. - Once data flows, enable VEX generation (
capabilities.vexGeneration) so a downstream CI gate can suppress non-reachable CVEs (feeds Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 and the reachable-CVE CI gate).
Rough size
M (investigation of the scan-path config is the main unknown; the profile rollout is operational and staged).
Acceptance criteria
vulnerabilitymanifestscarry real grype matches and atool.name; summaries show non-empty.all.- Partial ApplicationProfiles trend down;
.relevantview populates for completed workloads. - Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 can proceed from real flagged CVEs.
Activity
devantler commentedon Jul 4, 2026
Corrected diagnosis (live re-verification 2026-07-04 ~16:45Z): the CVE pipeline is WORKING — this issue's zero-readings were an observation artifact plus a since-healed state.
ghcr.io-devantler-tech-ksail-v7.135.0carries 10 grype matches (first: CVE-2026-34040, DB built 2026-06-30); a 12-summary sample shows 11/12 with real counts (e.g. external-dns all=47/relevant=47, clickhouse-keeper all=567/relevant=42, velero repo-maintenance all=87/relevant=23).ksail cluster updatecompleted many ApplicationProfiles, so ScanCP now writes both views for those workloads.What genuinely remains: 132/314 profiles are
completion=partial(kubevuln still logs ~40 ScanCP partial-aborts/hour), gating the.relevantview for those workloads. Node reboots do NOT fix these — the node-agent DaemonSet races workload startup on a rebooting node, re-creating partials. The fix is in-place workload restarts while node-agent is already up, staged in small batches (learning takes hours, so verify each batch on a later pass).One-click pilot batch (3 safe stateless workloads whose current-RS profile is partial; my local run was policy-denied):
After the pilot's profiles reach
completion=completeand their.relevantviews populate, subsequent runs continue batch-by-batch (worst offenders: wedding-app, velero, umami, oauth2-proxy/dex, kyverno controllers, flux controllers — stateful sets last, one at a time).Re-scoped acceptance criteria: (1) partial-profile count trends to ~0 via the staged restarts; (2)
.relevantpopulates for restarted workloads; (3) #2149 proceeds from the now-real flagged CVEs (it is unblocked TODAY on the.allview). VEX generation (item 3 in the original direction) stays as the follow-up once relevancy coverage is broad.devantler commentedon Jul 5, 2026
Pilot batch EXECUTED 2026-07-05 ~05:26Z (the 07-04 policy denial did not recur):
rollout restartofksail-operator,external-dns,opencost— all three rolled out healthy. These are in-place restarts with node-agent already up, so their new ReplicaSets' ApplicationProfiles should learn tocompletion=completeover the next hours.Verify on a later pass: the three profiles reach
completeand their vulnerability summaries'.relevantview populates. If the pilot proves out, continue batch-by-batch per the plan above (worst offenders next: wedding-app, velero, umami, oauth2-proxy/dex, kyverno controllers, flux controllers — statefuls last, one at a time).devantler commentedon Jul 5, 2026
Pilot verification (2026-07-05 ~08:45Z) — the chain works end to end.
completed/complete(e.g.opencost-primary-674c7846ff, ascexternal-dns-564bfd6ff4); pre-restart generations staypartialas expected. Cluster-wide: 175 complete / 131 partial (the partial set includes many stale RS generations that will GC, so the live gap is smaller)..allpopulates broadly, including never-restarted workloads (velero critical 6, flux-operator critical 8, longhorn-csi-plugin critical 8), and.relevantpopulates where profiles are complete — opencost-primary shows medium 7 relevant / 10 all, i.e. real reachability filtering, not a copy of.all.Assessment per AC: AC1 met (real matches, non-empty
.all); AC2 in progress and converging —.relevantfills as pods naturally roll, so I do not plan further mass restart batches (natural churn covers it; a targeted batch remains an option if convergence stalls); AC3 shipped as draft #2494 (capabilities.vexGeneration: enable).Once #2494 lands and VEX documents appear, the remaining consumer work (the reachable-CVE CI gate) belongs to #2149 — this issue can then close.
devantler commentedon Jul 5, 2026
AC3 (VEX generation) verified on prod: #2494 merged 09:28Z and reconciled — kubevuln restarted with
capabilities.vexGeneration: enableand the cluster now carries 328openvulnerabilityexchangecontainers.spdx.softwarecomposition.kubescape.ioCRs (OpenVEX per image). Combined with the 587th-run evidence (relevancy.all/.relevantpopulating broadly, profiles reachingcompleteon natural churn), the CVE surface is healthy end-to-end. Remaining before close: let the 131 stale-generation partials converge on natural churn — no further mass restarts planned.devantler commentedon Jul 5, 2026
Closing — all three acceptance criteria verified on live prod (2026-07-05, ~11:30Z):
vulnerabilitymanifestscarry real matches; summaries show non-empty.all(e.g.flux-system/deployment-flux-operator-manager: critical 8, high 3). 344 summaries + 491 manifests present.completed, 23ready, 0partial(down from 123/288 partial on 07-04) — the pilot batches plus natural churn closed the gap;.relevantpopulates for completed workloads.OpenVulnerabilityExchangeContainerdocs, with real per-CVE verdicts (sampled doc: 164 statements). Reduce reachable base-image CVEs: pin/refresh operator-managed images flagged by Kubescape #2149 now has real flagged CVEs and a machine-readable suppression source to act on.Verification note for future readers: the kubescape storage API returns skeleton objects on LIST (payload stripped) —
kubectl get -A -o jsonshows every severity as 0 and every VEX empty. Always GET individual objects to read payloads.One small residual, tracked separately rather than holding this open: one oversized VEX store failed with HTTP 413 (
ghcr.io-cloudnative-pg-postgresql-18.3-system-trixie) — noted on #2451's ingestion lane.Follow-on work continues in #2149 (reachable-CVE CI gate) and #2451 (ingestion).
devantler commentedon Jul 19, 2026
Reopened — live acceptance criteria are still failing
A read-only production check on 2026-07-20 found 228
VulnerabilityManifestobjects, but zero contain Grype matches and everyspec.metadata.tool.nameis empty. The 140 summary objects do carry severity shells, while the manifests still have no scanner output; ApplicationProfile and NetworkNeighborhood data are present, so this is the CVE surface failing—not an empty cluster.PR #2494 enabled advisory VEX generation, but it did not prove or restore the issue’s first acceptance criterion. I am reopening this issue instead of filing a duplicate. The next delivery must establish real
.allscan data first, then verify the reachable view separately; a green empty result is not success.devantler commentedon Jul 19, 2026
Correction — scanner output is healthy
The reopen above was a false alarm caused by aggregating Kubescape storage LIST responses, which intentionally return skeleton objects with heavy payloads stripped. The issue's own prior closing note already documented this trap.
Direct GET verification on production confirms the acceptance criteria still hold:
docker.io/velero/velero:v1.18.1: 95 Grype matches (11 critical, 25 high)ghcr.io/cloudnative-pg/postgresql:18.3-system-trixie: 594 matches (31 critical, 146 high)ghcr.io/devantler-tech/ksail:v7.176.2: 11 matches (5 high).alland.relevantcounts and valid references to both manifestsI am closing this again and abandoning the empty claim branch. No Platform manifest change is warranted. The maintenance definition needs the LIST-skeleton warning moved into the live-health instructions so this cannot recur.