Repository navigation
fix(kubescape): preserve large vulnerability scan writes #3955
Description
Problem
Coroot reports a real storage application error for the Backstage vulnerability manifest:
Create - save object failedwrite metadata: insert metadata: sqlite: clear bindings: interrupted- key
/spdx.softwarecomposition.kubescape.io/vulnerabilitymanifests/kubescape/ghcr.io-backstage-backstage-1.52.0-993611
The exact manifest is absent afterward, so this is data loss and must not be exempted. The same scan also submits 6,604,113-6,604,551 byte openvulnerabilityexchangecontainers payloads, which the current 1,000,000 byte kind limit rejects.
Root cause evidence
The compatibility image sets SQLite busy_timeout to 60 seconds, but the pinned v0.0.297 storage code obtains every REST connection with pool.Take(poolCtx) where poolCtx expires after five seconds. zombiezen/sqlite binds that context to the connection interrupt for its entire checked-out lifetime. A legitimate foreground write waiting longer than five seconds is therefore forcibly interrupted before the reviewed busy timeout can help, after the payload file has been written but before metadata is persisted.
Acceptance criteria
- Add a failing regression test proving the pool-acquisition timeout cannot interrupt an already acquired connection during a legitimate longer operation.
- Separate the five-second pool-acquisition budget from a finite, longer checked-out connection lifetime without unsafe manual
SetInterruptrebinding. - Apply the fix consistently to the v0.0.297 compatibility image and publish/pin a signed immutable revision.
- Raise only
storage.kindQueues.openvulnerabilityexchangecontainers.maxObjectSizeto a finite value above the observed 6.605 MB payload with documented headroom. - Keep unrelated storage kinds and failure modes fail-visible.
- Verify the Backstage vulnerability manifest and VEX resource are persisted on a fresh scan, storage logs remain clean, and Coroot reports no storage log error after the historical entry ages out.
Metadata
Metadata
Assignees
Labels
Type
Fields
Priority
Effort
Projects
- Status
Activity
devantler commentedon Oct 5, 2026
The implementation part of this issue has already been delivered. #3957 merged at signed commit
75f74568e74d6172f0cd5852c98e6d014ac43896, including the failing-then-passing checked-out-connection regression. The committed compatibility patch separates the five-second acquisition budget from a finite 65-second connection lifetime, and only the VEX object-kind ceiling is raised to 8,000,000 bytes.Fresh read-only production verification on 2026-10-05 confirms the immutable
.6image is running and its Deployment is ready at the observed generation. The exact Backstage vulnerability manifest named in this issue is present with payload and resource version 15. The loaded Coroot alert view contains no storage log-error alert.This establishes deployment and restoration of that named manifest, but it does not establish a new coherent scan, the paired VEX result, or complete vulnerability coverage. The coverage collector still reports 31 listed/stored summary version inconsistencies on #4263, plus missing and stale image results. Keep the final production acceptance criterion open; further work should verify fresh paired scan persistence rather than reimplement the already deployed connection-lifetime fix. No result was deleted or rewritten by these checks.
devantler commentedon Oct 5, 2026
Fresh readback after #4500 confirms storage deployment generation 52 is fully observed, with one of one updated, ready and available replica. The aggregated API is Available again, and the service has a ready endpoint. The Coroot readiness and API 503 warnings coincide with that rollout; component availability has recovered.
This does not establish coherent new scan results or repair the outstanding metadata/payload splits. Those acceptance gates remain open. No result was deleted or rewritten by this read-only verification.
chore: agent-claim #3955 nonce=9dc99e073775a4794576535a041dcae0
devantler commentedon Oct 7, 2026
The last criterion fails today: the Backstage VEX record is being refused again (#4633 fixes it)
Read from production on 2026-10-07, read-only:
status code 413for it and for no other image. The stored copy was last changed on 2026-10-02.interrupted, no failed save..7compatibility image, one ready replica, no restarts since 5 October.So the write-interruption half of this issue has held for the manifest across at least two fresh scans. The size half has regressed: the record grew from 6.60 MB to 8.55 MB in sixteen days and passed the limit that was set with 21% headroom.
#4633 raises that one limit to the 50 MB the vulnerability manifests already have, and makes the check on it refuse anything under twice the largest record seen. #4637 tracks the gap that let this go unnoticed: a refused write is only a warning in a log.
What closes this issue: after #4633 deploys, the first 01:12 UTC scan must store the VEX record (no
413, norequest entity too large, the record's content time moves). Not read here: the Coroot alert view; the storage log is the evidence for "no storage error".devantler commentedon Oct 8, 2026
Closing: the first scan after #4633 stored the Backstage VEX record
#4633 merged on 2026-10-07 and is deployed. Read from production, read-only, at 01:19Z on 2026-10-08, during the 01:12Z scan:
413from the scannerThe write-interruption half had already held across earlier scans (see the note above). Both halves of this issue now hold on a fresh scan, so it is closed as completed.
Two things this does not cover, each with its own issue: a refused write is still only a warning in a log (#4637), and the limit is checked against record growth only by the test added in #4633.