Repository navigation
fix(security): Kubescape control-level scan detail is 54 days stale while summaries stay current #3275
Description
🤖 Generated by the Agentic Engineer
Evidence
Kubescape's two configuration-scan surfaces disagree, and only one of them is current. Measured
against admin@prod on 2026-08-20:
| Resource | Count | Newest object | State |
|---|---|---|---|
workloadconfigurationscansummaries |
2358 | 2026-08-20T10:07:15Z (today) | current, fresh objects daily |
workloadconfigurationscans (detail) |
1542 | 2026-06-27T14:50:44Z | 54 days stale |
Every detail object falls in a single 2026-06-17 → 2026-06-27 window. Nothing has been written to
that surface since, while summaries have continued daily and carry real severity data — today's
newest reads {"critical":0,"high":0,"low":0,"medium":3,"unknown":0}.
The scan itself is healthy: the scheduler Job succeeded at 2026-08-20T09:12:30Z and the summaries
appeared ~55 minutes later. So this is not the outage described in #2933 — that one is resolved.
This is one surface of a working scanner having gone quiet.
Impact
workloadconfigurationscansummaries carries severity counts. workloadconfigurationscans
carries the control-level detail — which control failed, on which rule. Any analysis that needs
to name a control reads the detail surface, and therefore reads June data.
Concretely, rule-level questions like "is C-0211 still failing, and on which rules" cannot be
answered correctly today: the answer describes the cluster as it was two months ago. A control fixed
since then still reads as failing, and a control that started failing since is invisible. Posture
work that graduates "zeroed" controls to enforcement is exactly the consumer that would be misled.
This is the same class of defect as #2933 — a surface that looks populated while being wrong — which
is why it is Security rather than a Chore. 1542 objects is a healthy-looking count.
Expected behaviour
Either the detail surface is written on every scan alongside the summaries, or — if the platform
deliberately retains only summaries — that is stated, and anything reading detail objects checks
their age instead of trusting their presence.
Acceptance criteria
- Determine why detail objects stopped being written on 2026-06-27 while summaries continued:
deliberate retention/cleanup, a storage or RBAC change, or a genuine regression. - If it is a regression: detail objects are written on each scheduled scan, evidenced by an
object newer than the fix. - If it is deliberate: record it where a reader of the detail surface will find it, and state
what the supported source of control-level detail is instead. - Either way, a consumer of control-level detail can tell stale data from current data without
already knowing this — an age check, not a presence check.
Notes
Found while re-verifying #2933, which is a different failure on the summaries surface and is now
resolved. Read control detail with a single-object GET: a LIST strips .spec.controls, so a
list-based check reports empty content for every object regardless of freshness.
Part of #2933's family of posture-integrity defects.
Metadata
Metadata
Assignees
Labels
Type
Fields
Projects
- Status
Activity
chore: agent-claim #3275 nonce=31d4fac4eb0a40b53266f64214ece6a3
devantler commentedon Aug 20, 2026
Re-verified independently against
admin@prodon 2026-08-20T22:07Z, and advanced acceptancecriterion 1. The freeze reproduces, the premise survives a check it could have failed, and the
cause is narrowed to a single change — but not yet determined.
The freeze reproduces, and it is not an artifact of
creationTimestampworkloadconfigurationscans(detail)workloadconfigurationscansummariescreationTimestampalone could not settle this: an object updated in place keeps its originalcreation time, so a frozen-looking detail surface could equally have been a fresh one being rewritten.
It is not — checked on a single-object GET:
resourceVersion: 1and nomanagedFields— createdonce, never updated since;
resourceVersion: 2, so this storage layer does increment on in-place update, and a summariesobject created today carries
resourceVersion: 1.So the detail surface is frozen on two independent signals, with the control proving the signal is
capable of showing movement. Both surfaces start on the same day, so this is one surface of a
working scanner stopping — not a surface that never worked.
What is ruled OUT
controls with an overall compliance score of 83, and wrote its results. fix(security): a Kubescape framework-load failure still reports zero findings instead of failing #2933's cause is
fixed and stays fixed.
survive it, so nothing is deleting them — new ones simply stop being written.
Loading exceptions/Loaded exceptionswith no error.
own scan (and
ci.yamlalready warns never to read a posture number off the stored resources),and
kubescape-backlog-bridgeconsumes the summary kind. The exposed consumer is a human oragent doing rule-level analysis — which is exactly criterion 4's concern.
Strongest lead: a 12-minute causal window
The last detail object was written at 2026-06-27T14:50:44Z.
c199323d"fix(kubescape): enable riskAcceptance so SecurityException CRs are applied (#2298)" is authored
2026-06-27T15:02:48Z — 12 minutes later — and no detail object has been written since. The
surface's first object (2026-06-17T20:09:12Z) likewise coincides with
0928f65c, which enabledoffline scanning that day. Both boundaries of the surface's entire lifetime land on a
kubescape capability change.
That correlation is tight enough to lead with and not sufficient to call a cause: exceptions
load cleanly today, so if
riskAcceptance: enableis responsible it is by changing which write paththe scan takes, not by failing visibly.
Where the remaining question actually sits
helm-release.yamldocuments that an offline continuous scan "persists NOworkloadconfigurationscan CRs" because it dies before completing, and that the daily
keepLocalscheduler scan "DOES work". Today's scheduler scan completes and saves results — yet persists no
detail CRs while summaries continue. So the open question is narrow and specific:
On the current configuration, which component writes
workloadconfigurationscansummaries, and whydoes that path still run while the detail write does not?
Answering that decides criterion 1's branch (regression vs. deliberate), and everything after it
follows. Logs cannot answer it — the freeze predates every current pod's log window by weeks, which
is why this is reasoned from object metadata and change history instead.
Note on the staleness alert
The alert added in #3237 is deployed and working correctly — its most recent run reports
Posture scan data is stale: Newest stored scan is 2026-06-27T14:50:44Z — 54 day(s) old, across 1542 object(s). Detection is not the gap here; the gap is the cause.chore: agent-claim #3275 nonce=c35ca344d66e0c60d07b0831598dc5c3
chore: agent-claim #3275 nonce=56650cdd5c5c541d87da7cd300e81414
devantler commentedon Aug 28, 2026
Root cause is now confirmed and fixed in draft PR #3423 at
2a2dd71791139a3ea3d951a7496faa3104cc7c75.The scanner is pinned to v4.0.10. In a self-hosted deployment with no SaaS backend, that release can auto-enable submission, enter account-ID generation, and treat
chmod /home/nonroot/.kubescape: operation not permittedas fatal before the in-cluster report receiver persists detailed posture results. This explains the observed split: scans still compute a compliance score and the scheduler Job succeeds, while the detailed result surface remains stale. Risk-acceptance-triggered rescans reach the aborting request path because they do not carry the scheduler'skeepLocaloverride.Upstream fixed the full chain in kubescape/kubescape#2556: no-backend deployments skip SaaS initialization, explicit local-only intent is preserved, and permission tightening is best-effort rather than scan-fatal. The fix first ships in stable v4.0.12.
#3423 pins v4.0.12 without weakening the non-root security context, and adds an executable CI contract that rejects vulnerable or malformed scanner versions while preserving local-only scheduling and policy-artifact downloads. The regression was RED on v4.0.10 and is GREEN on v4.0.12; both repository overlays validate.
devantler commentedon Aug 28, 2026
The merged scanner upgrade removed the v4.0.10 local-mode abort and the exact recovery scan reached policy evaluation and saved results. That recovery exposed a second, independent persistence defect: the live scanner configuration has
continuousPostureScan: false.Kubescape v4.0.12 writes detailed
WorkloadConfigurationScanobjects only when that flag is true, while it always writes summary objects. This exactly matches production: summaries are current, but 1,542 detailed objects still stop at 2026-06-27.I am reopening this incident and preparing a follow-up fix that enables detailed persistence while rendering an empty continuous-scanning resource match set. The pinned operator source confirms an empty set creates zero Kubernetes resource watches, so scheduled scans can persist locally without restoring the event-triggered scan flood that led to the capability being disabled.
Production recovery remains in progress until a post-deploy scheduled scan makes the detailed objects current and the exact scan reaches a terminal success state.
16 remaining items
devantler commentedon Aug 29, 2026
Reopening because the live acceptance test after #3444 failed; that deployment slice auto-closed the issue before production recovery was proved.
The controlled post-deploy scan completed local evaluation, but foreground detailed-result persistence exceeded its request deadline. The detailed storage surface remained unchanged at 1,597 objects, newest
2026-08-28T21:16:45Z, with zero hydrated control maps.The immediate-transaction fix removed the two-maintenance-worker lock-promotion race, but it reserved SQLite's sole writer throughout minutes of profile reads. Background workers could hand off that reservation long enough to starve the foreground REST write. #3448 is now merged with the corrected design: retain deferred transactions, serialize profile maintenance to one worker, keep the 60-second busy timeout, and prove foreground write availability with an executable regression.
The signed immutable image is awaiting deployment through #3450. This incident remains open until that PR merges, Flux reconciles it, and a new controlled live scan advances the detailed-object timestamp with nonempty controls and no correlated persistence failure.
devantler commentedon Sep 2, 2026
Re-measured live today — the headline is no longer true, and the proxy behind it does not hold
This issue's evidence is from 2026-08-20. Re-measured against prod on 2026-09-02, read-only:
workloadconfigurationscansummariesworkloadconfigurationscans(detail)The two surfaces now carry the same newest timestamp. The detail surface is being written: 44 objects created in the last 24 h, 802 in the last 7 days. So "54 days stale, nothing written since 2026-06-27" is no longer the state of the cluster.
The June cohort persists — but it is not what it looks like
1541 objects still carry June creation stamps, all in the original 2026-06-17 → 06-27 window (the issue counted 1542). Two hypotheses, both tested:
Orphans left by deleted workloads? — NO. Sampled six June-cohort
Deploymentobjects; all six workloads still exist (actual-budget,ascoachingogvaner×2,backstage,cert-manager×2). They are live workloads, not residue.Frozen because never updated? — the evidence does not support that reading either. A June object carries
resourceVersion: "1", which looks conclusive until you run the control: an object created today at 07:01Z also carriesresourceVersion: "1", and across the whole surfaceresourceVersionvaries (1, 2, 3, 8, 9 …). Sorv: "1"means "not rewritten since it was created", which is equally true of a fresh object — it is not a staleness signal.Every object also carries a
kubescape.io/sync-checksumannotation, which is what a checksum-based sync looks like: an object is rewritten when the scanned workload's configuration changes, not on every scan. Under that model a June creation stamp on an unchanged workload is the expected steady state, not a defect — andcreationTimestampis not a freshness proxy for this surface at all. That proxy is what the original measurement rests on.What this does and does not settle
Settled: the detail surface is live, both surfaces agree on their newest object, and the June cohort is not orphaned.
Not settled — and this is the decisive test: whether a June object's content still describes its workload correctly. Take a June-cohort workload whose spec has demonstrably changed since June, single-object
GETits scan object (a LIST returns.spec.controlsnull, so a list sweep answers this wrongly), and compare the controls against the current spec. If they disagree, the sync is genuinely missing updates and this issue stands with a corrected diagnosis. If they agree, the surface is behaving as designed and the issue should close.I did not run that comparison — it needs a workload with a known spec change since June, which I could not establish read-only in the time available.
Recommendation
Do not "fix" this on the current evidence: the measurement it rests on does not distinguish a stale object from a correctly-unchanged one. Re-frame the acceptance criteria around content correctness rather than object age, then re-decide. Leaving this open and unclaimed with the corrected framing so the next run starts from the right question.
chore: agent-claim #3275 nonce=d827aa63fa3ce70f6bdfdc69bcdfd86c
devantler commentedon Sep 2, 2026
Ran the decisive test this issue asked for — the surface is behaving as designed
The previous comment named the test it could not run:
It blocked on "it needs a workload with a known spec change since June, which I could not establish
read-only". That gap is closable: a Deployment's rollout time is readable read-only from
status.conditions[type=Progressing].lastUpdateTime. 60 of the 64 June-cohort Deployment scanobjects belong to workloads that rolled out after their scan object was created.
The content is current — proven by a dated marker inside it, not by object age
Rather than compare control verdicts field-by-field (which needs a per-control oracle), I used a
dated marker:
k8s/bases/infrastructure/cluster-security-exceptions/pod-security-mutations-unscoped.yamlwas added 2026-08-18 (
c8c2197e, #3225 —git log --diff-filter=A, not-1, which returns thenewest commit and would have dated it 08-30).
Every June-cohort object cites it:
appliedIgnoreRules: [pod-security-mutations-unscoped/…]Content that cannot predate 2026-08-18 sits inside objects stamped 2026-06-17. The objects are
rewritten in place;
creationTimestamprecords when the CR was first created, never when itscontent was last written. That is a complete census of the cohort, not a sample.
Both other claims in this thread also re-test clean
detailnewest2026-09-02T10:21:01Z·summariesnewest2026-09-02T10:21:01Z— identicalcontinuousPostureScan: false(2026-08-28)truein the liveks-cloud-config, and deliberately declared inhelm-release.yamlThe original incident's root cause — the v4.0.10 local-mode abort — was fixed by the v4.0.12 pin in
#3423.
Decision: closing
Every premise this issue rests on has been re-measured and none survives. Per the previous comment's
own criterion, the agreeing case is the close case.
The reusable finding —
creationTimestampis not a freshness proxy for this surface. The sync ischecksum-driven: an object is rewritten when the scanned configuration changes, so an old creation
stamp on an unchanged workload is the designed steady state. A freshness monitor on this surface must
key on the newest object across the surface, or on dated content, never on per-object age.
Two method notes for whoever measures here next:
.spec.controlsasnull. Only a single-objectGETpopulates it, so a listsweep answers content questions wrongly.
resourceVersionon this surface is per-object (it starts at 1), not the cluster's global etcdcounter — but it is still not a staleness signal on its own, for the reason the previous comment
established.
Not re-litigating the separate, already-documented caveat that a suppressed control reports
status: passedwithsubStatus: "w/exceptions"— that fail-open is recorded in the exception fileitself and tracked in #2824.
Reopen if a content-level disagreement is ever demonstrated; object age alone is not that evidence.