Repository navigation
Conversation
…ring maintenance
✅ PR checks passedThe linked issue has a description and all required Shipping project fields set. Thanks! |
github-actions
Bot
added
backend
safe to test
![gitar-bot[bot]](https://avatars.githubusercontent.com/in/827041?s=60&v=4){kind=small}
| private CompactionResult compactHoldingSlot(final String slot, final String jobId) { | ||
| CompactionResult result = null; | ||
| try { | ||
| result = compactExclusively(jobId); | ||
| } catch (RdfReindexRunLock.HeldByAnotherRun reindexRunning) { | ||
| LOG.info("Skipping RDF compaction while a reindex runs; it compacts itself"); | ||
| } finally { | ||
| if (result == null || !result.succeeded()) { | ||
| locks.releaseLock(slot, jobId); | ||
| } | ||
| } | ||
| return result; | ||
| } | ||
|
|
||
| private CompactionResult compactExclusively(final String jobId) { |
There was a problem hiding this comment.
⚠️ Bug: Pause released while a timed-out/untracked Fuseki compaction still blocks
FusekiCompaction returns TIMED_OUT when maxWait runs out and UNKNOWN when an IOException cuts the poll short. CompactionResult.Outcome.TIMED_OUT itself says writes "stay blocked until it ends". In both cases the Fuseki task is still running. Even so, RdfLiveWritePause.during releases the cluster-wide pause as soon as compact() returns (RdfStorageMaintenanceService.java:153, RdfIndexApp.java:417-418). The queued live writes then reach a Fuseki writer that is still blocked, time out, and use up delivery attempts. With enough of them they get dead-lettered. That is the failure the ADR says this PR prevents. compactHoldingSlot also releases the RDF_STORAGE_NEXT_COMPACTION slot for a non-success result, so 15 minutes later a second compaction can be started against a dataset that is still compacting. Fix: when the outcome is TIMED_OUT or UNKNOWN with a task id, keep the pause held, or re-poll /$/tasks/{id} until the task is gone. Also don't release the slot until Fuseki reports no running Compact task.
Keep the pause until the task is no longer running when the wait ran out, and don't free the slot early.:
return rdf.withServingWritesPaused(jobId, () -> {
CompactionResult result = rdf.compact(maxDuration());
if (result.outcome() == CompactionResult.Outcome.TIMED_OUT) {
LOG.error("Compaction {} still running in Fuseki; keeping writes paused", result.taskId());
result = rdf.compact(maxDuration()); // or a dedicated awaitTask(taskId)
}
return result;
});
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
| try { | ||
| return new Run(serverBaseUrl, authenticate).awaitStarted(dataset, maxWait, requestedAt); | ||
| } catch (IOException exception) { | ||
| return CompactionResult.of( | ||
| CompactionResult.Outcome.UNKNOWN, null, describe(exception), requestedAt, clock); | ||
| } catch (InterruptedException exception) { | ||
| Thread.currentThread().interrupt(); | ||
| return CompactionResult.of( | ||
| CompactionResult.Outcome.UNKNOWN, | ||
| null, | ||
| "Interrupted while compacting; the task may still be running", | ||
| requestedAt, | ||
| clock); | ||
| } |
There was a problem hiding this comment.
💡 Edge Case: Non-JSON Fuseki reply makes compaction throw instead of reporting an outcome
FusekiCompaction.compact catches only IOException and InterruptedException. field() calls JsonUtils.readTree, which throws an unchecked JsonParsingException when a 200 reply has a body that is not JSON (for example an HTML page from a proxy, or a truncated body). That exception escapes compact(). RdfStorageInterface.compactStorage() is documented as best-effort and "never thrown", but it now delegates to compact(). The old implementation swallowed these errors. As a result, prepareBuildDataset (an unwrapped indexingRepository.compactStorage() call) and resetServingDataset now fail the whole rebuild on such a reply. Fix: handle parse failures in field()/TaskState.parse and map them to UNKNOWN, or catch RuntimeException in compact() and return UNKNOWN.
Treat unparseable bodies as having no field.:
static String field(final String body, final String name) {
if (body == null || body.isBlank()) {
return null;
}
final JsonNode value;
try {
value = JsonUtils.readTree(body).get(name);
} catch (JsonParsingException unparseable) {
LOG.warn("Unparseable Fuseki task response: {}", body);
return null;
}
return value == null || value.isNull() || value.asText().isBlank() ? null : value.asText();
}
- Apply fix
Check the box to apply the fix or reply for a change | Was this helpful? React with 👍 / 👎
✅ Generated Sources Auto-UpdatedThe generated TypeScript types ( |
…lure, metrics in OntologyMetrics
Jest test CoverageUI tests summary
|
…e' into fix/36437-rdf-storage-maintenance
Code Review
|
| Compact |
|
Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source
|
|
Describe your changes:
Fixes #36437
Problem
Fuseki's TDB2 store never reuses disk by itself. Every write copies the index pages it touches, and the old pages stay until the dataset is compacted. Rewriting one entity costs about 750 KB, so a store that only receives live writes grows by tens of gigabytes a night under ingestion, until the volume is full and Fuseki crashes on every restart.
Why it happened
Only the RDF reindex app compacted, weekly, and only after its
CLEAR ALL. That clear ran under the 60-second request timeout. On a large store it timed out, the run failed before compacting, and nothing ever freed space again. Fuseki also blocks writers for a whole compaction, so live writes that arrived meanwhile timed out, used up their retries and were dead-lettered.What this PR changes
RDF_COMPACTION_INTERVAL_HOURS), never during a reindex. A failed compaction is logged as an error and retried 15 minutes later.RDF_MAINTENANCE_MAX_DURATION_MINUTES, 240) instead of the request timeout. Compaction always follows; if the clear failed, the run then fails.success: true. A task Fuseki no longer reports counts as unknown, not success. Compactions are timed inontology.rdf.compaction, with a last-success gauge for alerts.The leases use the existing
rdf_reindex_locktable, so there is no migration. Decision record:ADR:2026-10-28-rdf-storage-maintenance-pauses-serving-writes-and-compacts-on-a-schedule.What operators will notice
Disk use stays near the live graph's size. Live writes wait out a compaction (about 20 seconds for a 1.2M-triple graph) instead of failing.
How it was tested
RdfStorageMaintenanceITon MySQL and PostgreSQL: paused writes wait without using attempts and apply afterwards, only the holder writes during a pause, an expired pause stops holding writes, and a maintenance clear plus compaction shrink the dataset. The existing RDF delivery, blue/green, lock and readiness ITs pass.Not in this PR
Type of change:
Checklist:
Fixes <issue-number>: <short explanation>Fixes #<issue-number>above.docs/decisions/2026-10-28-rdf-storage-maintenance-pauses-serving-writes-and-compacts-on-a-schedule.md.