Fixes #36437: Compact the RDF store on a schedule and pause writes during maintenance

bootstrap/sql/migrations/native/2.1.0/mysql/schemaChanges.sql

@@ -915,6 +915,30 @@ PREPARE rdf_projection_health_watermark_stmt FROM @rdf_projection_health_waterma
 EXECUTE rdf_projection_health_watermark_stmt;
 DEALLOCATE PREPARE rdf_projection_health_watermark_stmt;
 
+-- When storage maintenance last compacted the serving dataset, so a compaction that stops
+-- happening, for any reason, shows as a degraded projection instead of going unnoticed.
+SET @rdf_projection_health_compaction_ddl = (
+  SELECT IF(
+    EXISTS (
+      SELECT 1
+      FROM information_schema.columns
+      WHERE table_schema = DATABASE()
+        AND table_name = 'rdf_projection_health'
+        AND column_name = 'lastCompactedAt'
+    ),
+    'SELECT 1',
+    'ALTER TABLE rdf_projection_health ADD COLUMN lastCompactedAt bigint NOT NULL DEFAULT 0'
+  )
+);
+PREPARE rdf_projection_health_compaction_stmt FROM @rdf_projection_health_compaction_ddl;
+EXECUTE rdf_projection_health_compaction_stmt;
+DEALLOCATE PREPARE rdf_projection_health_compaction_stmt;
+
+-- Transfer RDF delivery to pause-aware consumers before producing pipelineExecution payloads.
+-- Deleting first waits for an old in-flight delivery; old producers still use the enqueue guard.
+DELETE FROM rdf_live_write_guard WHERE id = 'drain';
+INSERT IGNORE INTO rdf_live_write_guard (id) VALUES ('drain-v2');
+
 -- Credential verification joins user_entity to user_tokens on (userId, tokenType) and reads token.
 -- user_tokens had only PRIMARY KEY (token), so that predicate scanned every row, refresh and
 -- email-verification tokens included. Including token makes the fingerprint read index-only.

bootstrap/sql/migrations/native/2.1.0/postgres/schemaChanges.sql

@@ -681,6 +681,16 @@ ALTER TABLE rdf_active_dataset ADD COLUMN IF NOT EXISTS generation VARCHAR(36);
 ALTER TABLE rdf_projection_health
   ADD COLUMN IF NOT EXISTS enqueuedWatermark BIGINT NOT NULL DEFAULT 0;
 
+-- When storage maintenance last compacted the serving dataset, so a compaction that stops
+-- happening, for any reason, shows as a degraded projection instead of going unnoticed.
+ALTER TABLE rdf_projection_health
+  ADD COLUMN IF NOT EXISTS lastCompactedAt BIGINT NOT NULL DEFAULT 0;
+
+-- Transfer RDF delivery to pause-aware consumers before producing pipelineExecution payloads.
+-- Deleting first waits for an old in-flight delivery; old producers still use the enqueue guard.
+DELETE FROM rdf_live_write_guard WHERE id = 'drain';
+INSERT INTO rdf_live_write_guard (id) VALUES ('drain-v2') ON CONFLICT (id) DO NOTHING;
+
 -- A search that pins a memory anchor looks up the edges from the pinned ids into context memories
 -- (ADR:2026-10-09-search-admits-memories-of-pinned-anchors). The existing fromId/relation index
 -- does not cover toEntity, so a pinned domain or team reads all its HAS edges before filtering.

conf/openmetadata.yaml

@@ -984,6 +984,11 @@ rdf:
   federation:
     enabled: ${RDF_FEDERATION_ENABLED:-false}
     allowedEndpoints: ${RDF_FEDERATION_ALLOWED_ENDPOINTS:-[]}
+  # TDB2 keeps every page a write replaces until compaction; compact on a schedule with writes paused.
+  storageMaintenance:
+    enabled: ${RDF_STORAGE_MAINTENANCE_ENABLED:-true}
+    compactionIntervalHours: ${RDF_COMPACTION_INTERVAL_HOURS:-24}
+    maxDurationMinutes: ${RDF_MAINTENANCE_MAX_DURATION_MINUTES:-240}
 
 # Cache Configuration
 # Caching layer for entity metadata, relationships, and tag usage to reduce database load

docs/decisions/2026-10-28-rdf-storage-maintenance-pauses-serving-writes-and-compacts-on-a-schedule.md

@@ -0,0 +1,93 @@
+# RDF storage maintenance pauses serving writes and compacts the serving dataset on a schedule
+
+- **Status:** Accepted
+- **Revisions:** v1 2026-10-28 (initial), v2 2026-10-30 (task recovery and queue protocol fence)
+- **Deciders:** Platform
+- **Guard:** `RdfStorageMaintenanceIT`, `RdfStorageMaintenanceServiceTest`, `FusekiCompactionTest`,
+  `CompactionBlockedWriteTest`, `RdfIndexAppTest`, `RdfLiveWriteTest`, `RdfMaintenanceRecoveryIT`
+- **Related:** #36437
+
+## Context
+
+TDB2 copies every index page a write transaction touches and keeps the old pages until the dataset
+is compacted. Rewriting one entity costs several hundred kilobytes of disk, so a store that only
+receives live writes grows by tens of gigabytes a night under ingestion. Until this decision only the
+RDF reindex app compacted, and it compacted only after its `CLEAR ALL`, which ran under the 60-second
+request timeout. On a large store the clear timed out, the run failed, and nothing was compacted
+again, until the volume filled and Fuseki crashed on every restart.
+
+Fuseki also blocks every writer of a dataset for the whole compaction. A live write that reached
+Fuseki meanwhile timed out, used one of its ten delivery attempts, and after enough of them was
+dead-lettered, marking the projection degraded until a full rebuild.
+
+## Decision
+
+- One server compacts the serving dataset every `storageMaintenance.compactionIntervalHours` (24).
+  The slot is a lease, `RDF_STORAGE_NEXT_COMPACTION`, in `rdf_reindex_lock`, taken for one interval.
+  A compaction Fuseki never started releases it, so the next check (every 15 minutes) retries.
+  One Fuseki started and that did not succeed keeps it: it failed or restarted Fuseki because the
+  volume cannot hold the copy, or it is still copying, and retrying would only repeat that. A
+  failed copy also stays allocated until Fuseki restarts, so the error asks for a bigger volume and
+  a restart.
+- One rule makes a stopped schedule visible, whatever stopped it (failures, a reindex or inference
+  run holding its lock, a store that cannot be reached): each successful maintenance compaction is
+  recorded in `rdf_projection_health.lastCompactedAt`, and once none has succeeded for two intervals
+  every check logs an error and the RDF projection status reads `DEGRADED` until one does.
+  The health clock starts before the first compaction; scheduled health checks continue while the
+  compaction worker waits, so an unreachable admin API cannot hide a paused projection indefinitely.
+- Maintenance holds the RDF reindex and inference locks while it runs, so it never overlaps an
+  indexing or inference run. The first check runs five minutes after a server starts. Nothing
+  waits for a rolling upgrade: a compaction copies only the live data, so it is short, and a server
+  too old to know the pause just waits out a compaction shorter than its request timeout.
+- Before anything blocks Fuseki's writer, maintenance takes the write pause, a lease named
+  `RDF_LIVE_WRITES_PAUSED` in the same table, and waits for in-flight serving writes to finish.
+  `RdfDatasetManager.writeActive` then holds back every other writer. A live-write queue entry is
+  deferred at once, using no delivery attempt and not tripping the circuit breaker. Any other
+  writer, which nothing retries, first waits for the pause to end up to the RDF request timeout,
+  as long as Fuseki's own writer lock would have kept it waiting, and only then gets
+  `RdfStorageMaintenanceException`, which the REST API answers with 503 and `Retry-After`. The pause
+  expires unless renewed, so a server that stops mid-maintenance holds writes for at most five
+  minutes.
+- How a write meets the pause, and who owns its failure, travels with the storage handle a writer
+  is given (`RdfWriteDelivery`), not with the thread it runs on. The live-write queue's handle is
+  held back at once and its failures stay the queue's; every other handle waits and marks the
+  projection degraded on failure; the maintenance job writes through its own pause with the
+  handle the pause hands it, checked against the lease's job id.
+- The pause ends only on Fuseki's answer: a status poll that fails, including an unreadable JSON
+  response, counts as still running. The serving compaction keeps polling the same task after its
+  maintenance budget and logs the overrun; it never submits another compaction to wait for the first.
+  If the start response is lost or unreadable, it waits until a readable task list confirms no
+  compaction is running. Losing contact is not evidence that Fuseki stopped copying. A write
+  that times out while Fuseki reports a running compaction, whoever started it, is held back like a
+  paused write instead of counting as failed.
+- Pipeline run triples go through the live-write queue, so a maintenance window does not lose them.
+  The release migration first deletes the legacy `drain` guard, waiting for any old consumer already
+  holding it, then creates `drain-v2`. Upgraded consumers use this new fence; old consumers find no
+  legacy fence and stop before reading an unknown payload or consuming its delivery attempts. The
+  `enqueue` guard is unchanged, so old servers can still produce their supported commands during a
+  rolling upgrade. An upgraded RDF updater refuses to start without the new consumer fence.
+- After a blue/green promotion, the run empties and compacts the dataset it retired, which
+  nothing reads again, so between runs only the serving dataset holds a full copy. A build that
+  fails or is stopped is kept for inspection, and the next build clears it.
+- Compaction and the clear that starts an in-place rebuild run under
+  `storageMaintenance.maxDurationMinutes` (240), not the per-request timeout, and do not consult or
+  feed the circuit breaker. For a serving compaction this is a warning threshold: the copy cannot
+  be cancelled, and writes stay paused while its task is still running or cannot be read. Build and
+  retired datasets retain the bounded wait. A `CLEAR` cannot be cancelled in Fuseki, so a shorter
+  budget only abandons a clear that still completes.
+- An in-place rebuild clears, then always compacts, then reloads the ontologies. A clear that fails
+  is still followed by the compaction, and then fails the run.
+- A compaction counts as succeeded only when Fuseki's task reports `success: true`. A task Fuseki no
+  longer reports, or that is not a compaction, is `UNKNOWN`, never success.
+
+## Consequences
+
+- Disk use tracks the live graph plus at most one interval of growth, without operator action.
+- Maintenance delays live writes by the compaction's duration; it no longer loses them.
+- Any new code path that writes the serving dataset must go through `writeActive`, or it bypasses
+  the pause and may block on Fuseki during a compaction.
+- After the protocol migration, keep an upgraded RDF server available to drain the queue. An
+  old-only rollback leaves queued writes pending. Restore the legacy consumer fence only after the
+  new payloads have been delivered by an upgraded server, or after an explicit reconciliation from
+  authoritative metadata. Never enable both consumer fences together.
+- Revisit the interval if a deployment grows faster than its volume allows within one interval.

docs/rdf-production-setup.md

@@ -410,12 +410,13 @@ server stops mid-run (a rolling deploy, an OOM kill, a node drain), the graph st
 enough to overlap a deploy; an interrupted blue/green run leaves the previous graph serving. Enabling **Blue/Green Rebuild** in the RDF Indexing application's configuration
 (alongside *Recreate RDF Store*) changes the shape of a rebuild:
 the run builds into an idle second dataset and switches to it only after the build succeeds, so the
-previous graph keeps serving until cutover, and its dataset is retained until the next rebuild
-reuses it.
+previous graph keeps serving until cutover. Once the new dataset serves, the run empties and
+compacts the previous one, which nothing reads again. A run that fails or is stopped leaves its
+half-built dataset in place for inspection; the next rebuild clears it.
 
 Builds alternate between the fixed `openmetadata_a` and `openmetadata_b` names derived from the
-configured endpoint. The original `openmetadata` directory remains present too: allow space for
-up to three dataset copies plus compaction headroom until an operator retires the unused original.
+configured endpoint. The original `openmetadata` dataset is emptied the same way once a promotion
+retires it, so between runs only the serving dataset holds a full copy.
 Custom endpoint names need matching base, `_a`, and `_b` assembler declarations.
 
 The run records the selected dataset, rebuild generation, and payload budget on the job, and
@@ -451,26 +452,55 @@ idle dataset's disk entirely, stop Fuseki and remove the unused
 
 ## Compaction and disk growth
 
-OpenMetadata requests Fuseki compaction after clearing a recreate run and after indexing its
-records. Blue/green runs compact the target before promotion so live writers can continue using
-the serving dataset. In-place runs compact after reporting completion. TDB2 copies index pages
-even during insert-only writes; clearing and compacting an
-empty target does not reclaim the obsolete pages created during its subsequent rebuild. Larger
-bounded batches reduce transaction churn, but the volume must still accommodate peak usage before
-compaction. See the [scale validation](rdf-scale-validation.md) for measurements and workload settings.
-Compaction is best-effort: an indexing run can succeed even if disk reclamation fails.
-
-To compact manually:
+TDB2 never reuses space on its own. Every write transaction copies the index pages it touches,
+and the old pages stay on disk until the dataset is compacted. A store that only receives live
+writes therefore grows with every entity change: a full rewrite of one entity costs several
+hundred kilobytes, so a nightly ingestion that changes tens of thousands of entities adds tens of
+gigabytes. Only compaction returns that space, and it needs free space about the size of the live
+data to copy it into.
+
+OpenMetadata compacts the serving dataset itself:
+
+- **On a schedule.** One server compacts every `RDF_COMPACTION_INTERVAL_HOURS`, while the RDF
+  reindex lock keeps it from overlapping an indexing run.
+- **During indexing.** A recreate run clears the dataset, compacts it while it is empty, and
+  compacts again after loading. Blue/green runs compact the target before promotion and empty
+  the dataset the promotion retires.
+
+Fuseki blocks every writer to a dataset for the whole compaction, so OpenMetadata pauses its own
+writes to the serving dataset first, on every server. Queued live writes wait in SQL and are
+applied once the pause ends; waiting does not count as a failed delivery attempt and does not mark
+the projection degraded. Other writes wait for the pause for up to `RDF_REQUEST_TIMEOUT_MS`, then
+get HTTP 503 with `Retry-After`. A write that times out while Fuseki compacts, whoever started the
+compaction, is retried the same way. Maintenance never overlaps a reindex or an inference run, and a
+server's first check runs five minutes after it starts.
+`RDF_MAINTENANCE_MAX_DURATION_MINUTES` bounds how long a compaction, or the clear that starts a
+recreate run, may hold that pause. Both run under this budget rather than the per-request timeout:
+Fuseki cannot cancel a `CLEAR`, so a shorter budget would only abandon a clear that still completes.
+
+Size the volume for the growth between compactions on top of the live data. TDB2 memory-maps
+its files, so a write on a full volume does not fail cleanly: the JVM gets SIGBUS and exits on
+every restart.
+
+Each compaction is logged, and timed by result in `ontology.rdf.compaction`; alert on
+`ontology.rdf.compaction.last.success` falling behind the interval. Every server reports the
+cluster's last successful maintenance compaction there. Once none has succeeded for two intervals,
+whatever the reason, each check logs an error and `GET /api/v1/rdf/status` reports
+`projectionState: DEGRADED` until the next one succeeds. A compaction Fuseki could not start is
+retried at the next check. One it started and that did not succeed is logged as an error
+and waits for the next interval: when the volume has less free space than the live data, the copy
+fails or restarts Fuseki, and a retry would only do the same. A copy that failed stays allocated
+until Fuseki restarts, so expand the volume, then restart Fuseki. To compact by hand:
 
 ```bash
 curl -u admin:<password> -X POST \
   'http://localhost:3030/$/compact/openmetadata?deleteOld=true'
 ```
 
-Fuseki returns an asynchronous task identifier. Inspect active and completed tasks at `/$/tasks`
-and confirm the data volume has enough space for both the old and replacement datasets. TDB2 index
-growth can reflect small transactions, failed or skipped compaction, or a volume that filled before
-compaction completed. [Jena storage FAQ](https://jena.apache.org/documentation/tdb/faqs.html).
+Fuseki returns an asynchronous task identifier; `/$/tasks/<id>` reports `success` once it has
+finished. Writes to the dataset wait until it does. See the
+[Jena storage FAQ](https://jena.apache.org/documentation/tdb/faqs.html) and the
+[scale validation](rdf-scale-validation.md) for measurements.
 
 ## Configuration reference
 
@@ -509,6 +539,9 @@ The OpenMetadata server reads the following settings from `conf/openmetadata.yam
 | `RDF_STRICT_OWL_PROFILE` | `true` |
 | `RDF_AI_ENABLED` | `false` |
 | `RDF_FEDERATION_ENABLED` | `false` |
+| `RDF_STORAGE_MAINTENANCE_ENABLED` | `true` |
+| `RDF_COMPACTION_INTERVAL_HOURS` | `24` |
+| `RDF_MAINTENANCE_MAX_DURATION_MINUTES` | `240` |
 
 Use `RDF_ENDPOINT` for new deployments. `RDF_REMOTE_ENDPOINT` remains a deprecated fallback for
 backward compatibility, and `RDF_ENDPOINT` takes precedence when both are set. Override the