[manerow]

Describe your changes:

Fixes #36437

Backport of PR #36472 to the 2.1 release branch, on top of PR #36473, the 2.1 backport of PR #36471. The commit applies unchanged. PR #36472 explains the problem.

What this PR changes

In the Fuseki image (docker/rdf-store; open-metadata/jena gets the same code for production):

• A write guard. Once the data volume has less free space than OPENMETADATA_STORAGE_MIN_FREE (10% by default), every write gets HTTP 507 Insufficient Storage and nothing is applied. Reads and compaction keep working.
• A storage report. GET /$/openmetadata/storage (admin only) returns the volume's size, free space and reserve, and whether writes are refused.
• No core dumps. The JVM no longer writes a heap-sized core dump when it crashes, even when a deployment sets its own JVM_ARGS.

In the server:

• Refused writes wait instead of failing. A 507 is a maintenance hold-back: queued writes wait without using a delivery attempt or tripping the circuit breaker. The queue's lag still marks the projection degraded, because a full volume needs attention.
• Urgent compaction. When a maintenance check (every 15 minutes) finds free space below RDF_URGENT_COMPACTION_FREE_PERCENT (50) or the guard refusing writes, it compacts without waiting for the daily slot. Compaction copies the live data, so 50% leaves room for the copy. If compacting cannot get back above the threshold, data compaction cannot remove fills the volume: the server logs at ERROR that it needs expanding, and urgent compactions stop for one interval while the daily one still runs.
• A recreate reindex recovers a full volume. Its storage check accepts a refused probe write, and when the guard refuses its clear it compacts to make room and clears once more.
• A store without the report, such as today's production image or stock Fuseki, keeps the daily schedule only.

The image-server contract is recorded in ADR:2026-10-28-the-fuseki-image-refuses-writes-below-a-free-space-reserve-and-reports-its-volume.

How it was tested on 2.1

• The commits are PR Fixes #36437: Refuse RDF writes before Fuseki's volume fills and compact early when it runs low #36472's, cherry-picked unchanged.
• 2.1 passes the RDF unit tests, including the storage report contract tests, and the targeted RDF integration tests on PostgreSQL, which build their Fuseki image from this branch's docker/rdf-store.
• Roll out the new Fuseki image with or after the OpenMetadata release that has this PR: an older server dead-letters the writes the guard refuses.

Type of change:

• Bug fix

Checklist:

• I have read the CONTRIBUTING document.
• My PR title is Fixes <issue-number>: <short explanation>
• My PR is linked to a GitHub issue via Fixes #<issue-number> above.
• I have commented on my code, particularly in hard-to-understand areas.
• For JSON Schema changes: no migration is needed; the new setting has a default.
• I have added tests (unit / integration) and listed them above.
• Decision check: docs/decisions/2026-10-28-the-fuseki-image-refuses-writes-below-a-free-space-reserve-and-reports-its-volume.md.

Release-stack refresh 73e9f27c315: merged the refreshed #36473, including the release branch's LDAP test fix. Spotless and the backend package build passed; StorageGuardRefusalTest, StorageFactsTest, and RdfStorageMaintenanceServiceTest passed (26 tests, 0 failures). The maintenance/delivery/storage Java sources match the tested main guard stack. Merge #36473 first, then retarget this PR to 2.1 and rerun Collate compatibility.

Refreshed maintenance base

Includes the recovery fixes from #36473: the shared serving pause survives a compaction budget overrun and malformed/unavailable task responses; the versioned consumer fence protects pipelineExecution during rolling upgrades; health reporting continues while compaction waits. Merge conflicts preserved both the disk-reporting/507 guard and the pause-preserving compaction API.

Validation at 7b956807ee6: 113 focused unit tests plus 10 recovery integration cases passed, using real MySQL and PostgreSQL with an HTTP Fuseki boundary. Coverage includes disk facts, HTTP 507 retry classification, urgent compaction, pause retention, legacy-consumer exclusion and migration handoff, and compaction-health reporting. Spotless and independent review passed; no unanswered important findings.

Merge #36473 first, then retarget this PR to its release branch and rerun the Collate build. The current feature-branch base does not exist in the Collate repository. Deploy upgraded servers before publishing/rolling out the guarded Fuseki image.

The broader affected-test run executed 620 service tests and stopped at the same 5 pre-existing RdfBatchProcessorTest failures caused by static registry interference with RdfResourceTest. It did not reach MCP or the wider IT matrix. The focused unit suite and 10 real-database recovery ITs above passed separately. The planner's ForwardingRdfStorage reference warning concerns additive pass-through delegation; combined storage behavior is exercised by the recovery ITs.

[github-actions]

✅ PR checks passed

The linked issue has a description and all required Shipping project fields set. Thanks!

[gitar-bot]

⚠️ Bug: Live data above 50% of the volume makes compaction run hourly forever

urgentSlot reserves an urgent compaction whenever freePercent() < urgentFreeSpacePercent (default 50). Take a healthy store whose compacted live data uses 55% of its volume: the store is steady and the guard (10%) is far from engaging, yet every urgent compaction leaves it at about 45% free. freedEnough returns false, so the slot is held only for URGENT_MIN_GAP, and then the cycle repeats. The result is a full compaction every hour, each one running withServingWritesPaused for up to maxDurationMinutes (240 by default). On large stores writes would be paused most of the time, where before this PR compaction ran once a day. The ERROR log tells you to expand the volume, but in the meantime the system behaves worse than without the feature. One fix is to back off further after a non-freeing compaction (for example, up to the regular interval) unless the guard is actually refusing writes. Another is to only treat the store as urgent when free space has dropped since the last compaction.

When a compaction cannot free space, only an engaged guard counts as urgent again before the regular interval:

private boolean needsSpace(final StorageFacts facts) {
  return facts.lowOnSpace() || facts.freePercent() < config.getUrgentFreeSpacePercent();
}

// in urgentSlot: hold the slot for compactionInterval() unless facts.lowOnSpace()
final Duration gap = urgent.map(StorageFacts::lowOnSpace).orElse(false) ? URGENT_MIN_GAP : compactionInterval();
final boolean reserved = urgent.isPresent() && reserve(URGENT_COMPACTION_KEY, jobId, gap);

_{Was this helpful? React with 👍 / 👎}

[gitar-bot]

💡 Quality: Bulk reconcile still logs guard refusals at ERROR

Every single-write path now calls rethrowIfHeldBack before it logs and wraps the exception. executeReconcileUpdate, which serves bulkStoreEntities, does not. While the guard refuses writes, each reconcile batch logs "Failed to bulk-store … entities in Fuseki" at ERROR with a full stack trace and wraps the RdfStorageLowDiskException. Deferral still works, because find walks the cause chain. But this goes against the stated contract that held-back writes "pass through unwrapped and unlogged", and the log fills with false errors during a low-disk period. The fix is to call rethrowIfHeldBack(e) first in that catch block.

In executeReconcileUpdate's catch block:

} catch (Exception e) {
  rethrowIfHeldBack(e);
  LOG.error("Failed to bulk-store {} entities in Fuseki", requests.size(), e);
  throw new RuntimeException("Failed to bulk-store entities in RDF", e);
}

_{Was this helpful? React with 👍 / 👎}

[github-actions]

✅ Generated Sources Auto-Updated

The generated TypeScript types (src/generated/) and dereferenced JSON
schemas (src/jsons/, public/jsons/) have been automatically updated
based on JSON schema changes in this PR.

[github-actions]

Jest test Coverage

UI tests summary

Lines	Statements	Branches	Functions
	80.17% (132500/165257)	66.77% (46944/70298)	69.83% (26615/38113)

[github-actions]

✅ Generated Sources Auto-Updated

The generated TypeScript types (src/generated/) and dereferenced JSON
schemas (src/jsons/, public/jsons/) have been automatically updated
based on JSON schema changes in this PR.

[gitar-bot]

Code Review ⚠️ Changes requested 1 closed / 3 findings

🟡 Medium risk · The Fuseki disk guard defers RDF writes and triggers urgent compaction; if compaction cannot free space, queued writes may remain stalled and projections degraded.

Adds a free-space write guard to the Fuseki image, a storage report endpoint, urgent compaction in the server, and reindex recovery for full volumes, with core dumps disabled when JVM_ARGS is overridden. Changes are requested: the urgent compaction threshold would trigger a full compaction hourly on stores whose live data exceeds 50% of the volume, pausing writes most of the time, and bulk reconcile logs guard refusals at ERROR with a stack trace instead of deferring them quietly. The core-dump guard gap was resolved.

⚠️ Bug: Live data above 50% of the volume makes compaction run hourly forever

📄 openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:143-153 📄 openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:177 📄 openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:185-196 📄 conf/openmetadata.yaml:990

urgentSlot reserves an urgent compaction whenever freePercent() < urgentFreeSpacePercent (default 50). Take a healthy store whose compacted live data uses 55% of its volume: the store is steady and the guard (10%) is far from engaging, yet every urgent compaction leaves it at about 45% free. freedEnough returns false, so the slot is held only for URGENT_MIN_GAP, and then the cycle repeats. The result is a full compaction every hour, each one running withServingWritesPaused for up to maxDurationMinutes (240 by default). On large stores writes would be paused most of the time, where before this PR compaction ran once a day. The ERROR log tells you to expand the volume, but in the meantime the system behaves worse than without the feature. One fix is to back off further after a non-freeing compaction (for example, up to the regular interval) unless the guard is actually refusing writes. Another is to only treat the store as urgent when free space has dropped since the last compaction.

When a compaction cannot free space, only an engaged guard counts as urgent again before the regular interval

private boolean needsSpace(final StorageFacts facts) {
  return facts.lowOnSpace() || facts.freePercent() < config.getUrgentFreeSpacePercent();
}

// in urgentSlot: hold the slot for compactionInterval() unless facts.lowOnSpace()
final Duration gap = urgent.map(StorageFacts::lowOnSpace).orElse(false) ? URGENT_MIN_GAP : compactionInterval();
final boolean reserved = urgent.isPresent() && reserve(URGENT_COMPACTION_KEY, jobId, gap);

💡 Quality: Bulk reconcile still logs guard refusals at ERROR

📄 openmetadata-service/src/main/java/org/openmetadata/service/rdf/storage/JenaFusekiStorage.java:942-952

Every single-write path now calls rethrowIfHeldBack before it logs and wraps the exception. executeReconcileUpdate, which serves bulkStoreEntities, does not. While the guard refuses writes, each reconcile batch logs "Failed to bulk-store … entities in Fuseki" at ERROR with a full stack trace and wraps the RdfStorageLowDiskException. Deferral still works, because find walks the cause chain. But this goes against the stated contract that held-back writes "pass through unwrapped and unlogged", and the log fills with false errors during a low-disk period. The fix is to call rethrowIfHeldBack(e) first in that catch block.

In executeReconcileUpdate's catch block

} catch (Exception e) {
  rethrowIfHeldBack(e);
  LOG.error("Failed to bulk-store {} entities in Fuseki", requests.size(), e);
  throw new RuntimeException("Failed to bulk-store entities in RDF", e);
}

✅ 1 closed

✅ Bug: Core-dump guard is lost whenever JVM_ARGS is overridden, which every manifest does

📄 docs/decisions/2026-10-28-the-fuseki-image-refuses-writes-below-a-free-space-reserve-and-reports-its-volume.md:41-42
This commit removes ulimit -c 0 from docker/rdf-store/entrypoint.sh. Core dumps are now turned off only by -XX:-CreateCoredumpOnCrash inside the Dockerfile's default ENV JVM_ARGS. Setting JVM_ARGS replaces that default instead of adding to it, and every manifest we ship that runs this image sets it. docker/rdf-store/kubernetes/fuseki-deployment.yaml:75-78 sets -Xms4g -Xmx4g -Xlog:gc*..., and docker-compose-rdf.yml:106 and docker/development/docker-compose-fuseki.yml:93 set their own heap flags, all without the flag. docs/rdf-production-setup.md:261 also tells operators to put properties in JVM_ARGS. So in those deployments a JVM crash writes a heap-sized core dump again. That is the failure the ADR says filled the node's disk and got the pod evicted, and the ADR still says the images prevent it. Fix it in a place users don't override: restore ulimit -c 0 in the entrypoint or add the flag through JAVA_TOOL_OPTIONS. Otherwise, add the flag to the k8s and compose JVM_ARGS values.

🤖 Prompt for agents

Code Review: Adds a free-space write guard to the Fuseki image, a storage report endpoint, urgent compaction in the server, and reindex recovery for full volumes, with core dumps disabled when JVM_ARGS is overridden. Changes are requested: the urgent compaction threshold would trigger a full compaction hourly on stores whose live data exceeds 50% of the volume, pausing writes most of the time, and bulk reconcile logs guard refusals at ERROR with a stack trace instead of deferring them quietly. The core-dump guard gap was resolved.

1. ⚠️ Bug: Live data above 50% of the volume makes compaction run hourly forever
   Files: openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:143-153, openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:177, openmetadata-service/src/main/java/org/openmetadata/service/rdf/maintenance/RdfStorageMaintenanceService.java:185-196, conf/openmetadata.yaml:990

   `urgentSlot` reserves an urgent compaction whenever `freePercent() < urgentFreeSpacePercent` (default 50). Take a healthy store whose compacted live data uses 55% of its volume: the store is steady and the guard (10%) is far from engaging, yet every urgent compaction leaves it at about 45% free. `freedEnough` returns false, so the slot is held only for `URGENT_MIN_GAP`, and then the cycle repeats. The result is a full compaction every hour, each one running `withServingWritesPaused` for up to `maxDurationMinutes` (240 by default). On large stores writes would be paused most of the time, where before this PR compaction ran once a day. The ERROR log tells you to expand the volume, but in the meantime the system behaves worse than without the feature. One fix is to back off further after a non-freeing compaction (for example, up to the regular interval) unless the guard is actually refusing writes. Another is to only treat the store as urgent when free space has dropped since the last compaction.

   Fix (When a compaction cannot free space, only an engaged guard counts as urgent again before the regular interval):
   private boolean needsSpace(final StorageFacts facts) {
     return facts.lowOnSpace() || facts.freePercent() < config.getUrgentFreeSpacePercent();
   }
   
   // in urgentSlot: hold the slot for compactionInterval() unless facts.lowOnSpace()
   final Duration gap = urgent.map(StorageFacts::lowOnSpace).orElse(false) ? URGENT_MIN_GAP : compactionInterval();
   final boolean reserved = urgent.isPresent() && reserve(URGENT_COMPACTION_KEY, jobId, gap);

2. 💡 Quality: Bulk reconcile still logs guard refusals at ERROR
   Files: openmetadata-service/src/main/java/org/openmetadata/service/rdf/storage/JenaFusekiStorage.java:942-952

   Every single-write path now calls `rethrowIfHeldBack` before it logs and wraps the exception. `executeReconcileUpdate`, which serves `bulkStoreEntities`, does not. While the guard refuses writes, each reconcile batch logs "Failed to bulk-store … entities in Fuseki" at ERROR with a full stack trace and wraps the `RdfStorageLowDiskException`. Deferral still works, because `find` walks the cause chain. But this goes against the stated contract that held-back writes "pass through unwrapped and unlogged", and the log fills with false errors during a low-disk period. The fix is to call `rethrowIfHeldBack(e)` first in that catch block.

   Fix (In executeReconcileUpdate's catch block):
   } catch (Exception e) {
     rethrowIfHeldBack(e);
     LOG.error("Failed to bulk-store {} entities in Fuseki", requests.size(), e);
     throw new RuntimeException("Failed to bulk-store entities in RDF", e);
   }

Verification

Rules No rules evaluated

Functional validation Not enabled

Set up

Auto-approval Not enabled

Set up

_{For more verification details, comment gitar display:verbose.}

Options

Display: compact → Showing compact verification results.

Comment with these commands to change the behavior for this request:

Compact

gitar display:verbose

_{Was this helpful? React with 👍 / 👎 | Powered by Gitar — free for open source}

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ui'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud

[sonarqubecloud]

Quality Gate passed for 'open-metadata-ingestion'

Issues
0 New issues
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.0% Duplication on New Code

See analysis details on SonarQube Cloud