perf(graph): bound retained Grafeo generation opens below cgroup memory budget

[ScriptedAlchemy]

Defect

A live Hotpath allocation run against PR #707's cgroup-aware/streaming binary (5de159b26) identified the dominant remaining memory cost: eager Grafeo generation opens.

Artifact: /fast/tmp/td-707-hotpath-alloc-report/hotpath-alloc.json
SHA-256: c3ec5d0dac658e4df19b7f446ff887a3a1ff3b454cde6de05ae00d780046ccc6

The alloc-bytes report recorded:

• graph_db.generation.open.engine: 5 calls, 29.9 GB exclusive allocation total (~6 GB/open).
• Total process allocation during the 688.56 s capture: 129.6 GB.
• Retained graph census was approximately 6.884 GB per opened generation.
• Live cgroup peak was approximately 27.9 GiB with ~13.3 GiB swap despite the cgroup-aware pressure mitigation.

The earlier interactive-catalog streaming fix removed one large duplicate materialization, but the live peak barely moved because multiple graph generations are still opened eagerly and retained concurrently.

Required outcome

• Attribute every production owner responsible for the five generation opens and distinguish active serving, warming, recovery, stale, and retirement ownership.
• Do not raise memory limits. Make non-serving generations lazy, hibernated, disk-backed, or otherwise bounded so retained graph owners fit under the effective cgroup budget.
• Keep exact serving and generation identity truthful; no empty-success fallback and no eviction of a leased serving generation.
• Add a scale test with multiple retained generations that asserts peak/retained bytes are bounded by the configured effective-memory ceiling.
• Expose per-generation retained bytes and owner/lease blockers in runtime status so pressure decisions are falsifiable.
• Re-run the same Hotpath allocation journey and show that graph_db.generation.open.engine no longer allocates roughly one full in-memory graph per retained generation and that RSS/swap stay below the cgroup high watermark during convergence.

[ScriptedAlchemy]

Slice 2 of the memory work landed on the #707 branch as 7638233 (perf(graph-db): release sealed generation staging rows once installed), picked up from the Cursor memory lane and re-verified: graph-db unit and contract suites (including the new staging_footprint tests: rows held for at most the active and in-flight generations, remount after release serves from the sealed store, hibernated retirement and release sweeps never open the engine), store-runtime unit suite, daemon maintenance and graph-replay tests, clippy and rustfmt on the pinned toolchain.

What it changes: a generation whose sealed store is installed, whose head is verified and which nothing depends on has its staging rows released after seal-install and serves from the sealed store; the release sweep keeps rows for in-flight, dependency-bearing and not-yet-sealed generations; a hibernated staging engine returns RetentionPending / StagingEngineHibernated instead of being opened for retirement or release; a zero-row leftover staging commit is treated as absent on recovery.

Not yet done for this issue's acceptance: the re-run of the allocation / RSS journey with the numbers in a closing comment. The measurement scaffolding that lane used (temporary systemd drop-ins pointing the operator daemon at a lane binary and gating starts on a WINDOW_OPEN marker) was removed so the production daemon runs again; a measurement window needs to be re-established deliberately before that evidence can be captured. Leaving open for the measurement.

[ScriptedAlchemy]

Live baseline on the operator profile with daemon beta.37+150a104b5, generation 27 (4,930 files / 308,529 chunks): RSS rose from 4.90 GB before code-graph projection to 21.97 GB while sealing/opening the 2.04 GB graph, crossing the 18.84 GB admission high watermark and the 20.94 GB configured limit. The graph reached code_graph_serving=ready, but the scheduler refused the next reconcile with requested=6442450944 and continued serving stale master. Profile was preserved; this is direct runtime evidence that the remaining blocker is the graph build/open resident peak, not a corrupt profile. Keep open pending a post-fix run below the existing watermark.

[ScriptedAlchemy]

Post-fix live acceptance still fails on exact pushed commit 250c5e0. Using the preserved operator profile, generation 29 stayed pending and continued serving stale master while process RSS climbed from about 0.42 GB at daemon start to 23.46 GB at 169 seconds and 26.26 GB at 221 seconds. I stopped the service through its normal daemon lifecycle before another OOM; systemd reports the prior cycle peaked at 26.0 GB. The runtime status response itself lagged badly during the spike, reporting a 1.80 GB sample from uptime 24 seconds while the live process was already 23.46 GB at uptime 169 seconds. This proves the retained-engine hibernation commit does not bound the active graph projection/build peak below the existing 18.84 GB watermark. Profile and durable data remain preserved. Keeping open.

[ScriptedAlchemy]

Attribution correction for the 250c5e0 generation-29 run: journal evidence proves this new 26.26 GB peak occurred before any project graph_engine_opened, sealed-store, or graph-publication event. Shutdown cancelled code_index background text projection and reported published text projection failed before graph seating. Therefore this run is a pre-seat text/index projection peak, distinct from the earlier generation-27 relational graph open peak. The currently dirty #830 publish-boundary staging release and malloc_trim work cannot fix this exact pre-publication peak. I am tracing the text projection allocation path separately and will file/link a focused child if it is not already #800 scope.

[ScriptedAlchemy]

Landed on the #707 branch (not merged to master yet):

• 250c5e01b perf(graph-db): hibernate retained sealed generation engines — sealed per-generation readers are lazy, released after the digest proof, idle peers stepped down on each install; the marker-hit proof path no longer materializes the graph; staging rows of sealed generations are released inline inside the permit; staging hibernation keeps sealed readers instead of uninstalling them.
• 298424812 perf(store-runtime): release publish memory at the build permit — malloc_trim at the permit boundary; measured marginal peak growth per published scope 0.59-0.61 GB → 0.21 GB on the 600-file/7,200-function corpus (see perf(store-runtime): release publish-time memory when the build permit is released #830).
• b22ec93d9 fix(graph-db): release sealed staging rows from the relational head — the registry sweep no longer requires an installed lease or an open engine, so the legacy per-generation rows (15 sealed generations, 8.6 GB container on the operator store) can drain.

Live trace that motivated this (operator daemon, 2026-09-04): the 26 GB RSS was grafeo_core lpg block::read_blocks deserializing the whole staging container into heap on every memory-graph open, reached from RetainedVerifiedGraphRuntimeV1::publish_verified_manifest; after rebuilding that store the same repo re-indexed at 4.5-6.7 GB. Remaining: whether an already-bloated container shrinks after row release (compaction of the staging container is still unresolved), a periodic sweep calling hibernate_idle_sealed_generation_engines, and the Hotpath allocation journey re-run on the fixed binary.

[ScriptedAlchemy]

Live confirmation on the operator daemon (build f4c8a1a): the sealed-row sweep now runs on hibernated engines and the agent-bundle project's staging container went from 10.15 GB to 5.60 GB within minutes of the first project open, with graph_staging_rows_released events for its sealed generations.

One residual class, logged as graph_staging_rows_retained stage="sweep" reason="no_verified_lease": projections under the retired code-scope: layout whose per-generation head was already retired by the #836 migration (graph_verified_heads_v1 no longer names them; only their graph_publication_replay_v1 rows remain). If their head was retired while the staging engine was hibernated, delete_generation_contents answered RetentionPending and their rows are still in the container, and the sweep has no relational head to bind a release to. The registry needs a rule for "replay exists, head retired, sealed artifact present": release against the replay's expected_recovered_digest when no other projection depends on the generation. On the rebuilt tracedecay store this is only log noise (the container is 28 KB); on stores that were never rebuilt it bounds how far the container can shrink.

[ScriptedAlchemy]

Follow-up on the residual RSS after the hibernation work: with the sealed readers lazy, the next largest resident consumer on the live daemon shape was allocator retention rather than graph state. glibc held 4.9–5.6 GB of freed-but-retained arena pages across a cold index of the tracedecay checkout (details and the per-phase heap split in #852). 875698e on the #707 branch registers malloc_trim as the last pressure reclaimer and samples RSS every 30 s so the reclaimers actually run during an index; no memory limit was raised.

[ScriptedAlchemy]

Residual reader-retention fix is now reviewed and published on codex/integrate-sept4: 5c2b44a retries sealed-reader hibernation, 786f083 makes reaping reader-safe by acquiring the real child database write lock, and 9382187 requires retired legacy cleanup proof. Exact lifecycle, real snapshot-reader, child-guard, full retention-contract, full library, and strict Clippy receipts are cc-4753, cc-4758, cc-4763, cc-4764, cc-4765, and cc-4766; independent review was clean. Keep #799 open for its broader live RSS/proportional retained-generation acceptance and the corrected post-trim pressure-state deployment proof.

[ScriptedAlchemy]

Merged down onto PR #707 at efce469 (merge commits e042f3b codex/integrate-sept4, 4bef15a codex/issue-863-ci-isolation-final, d0da2f4 codex/fix-post-trim-pressure-state, efce469 codex/port-753-activation-net). The combined tree passes cargo check --workspace --all-targets --locked (hauler cc-5021, 0 errors, 0 warnings); the workflow conflict between the two Windows-archive copies was resolved toward the 863 lane. Hosted CI on this tip is the next acceptance signal.

[ScriptedAlchemy]

Closed by central PR #707 at efce469: hibernated sealed readers (250c5e0), the sealed-row sweep on hibernated engines, the residual reader-retention fixes (5c2b44a, 786f083, 9382187) and the allocator trim under RSS pressure (875698e) are all central; no memory limit was raised. The remaining live-data share of the pre-seat working set stays tracked under #852.