Semantic vector publication times out under the fixed 30-second graph deadline

[HashemKhalifa]

tracedecay version

0.1.0-beta.37+1576b68e3df520e2113170518a98de97a31fdda8, built from source on macOS.

The same publication boundary is still present at PR #707 commit a5002ff.

What happened?

After the stale-stage recovery covered by #792 allowed a retained semantic vector generation to resume, staging advanced to 161,492 of 168,659 chunks. Final publication then failed in publish_generation with a graph operation deadline error, and semantic runtime returned to degraded/runtime_failure.

This is downstream of #792. Commit a5002ff fixes stale occupancy at begin_generation, but publish_generation_records still creates one authority with Instant::now() + GRAPH_OPERATION_DEADLINE where GRAPH_OPERATION_DEADLINE is 30 seconds. That same authority is reused across both:

1. prepare_publication_from_staged_native
2. publish_ready_stage

The prepare phase calls recovered_generation_digest_from_database, which decodes and hashes the full staged generation. Publication then performs additional whole-generation work under the remaining portion of the same deadline.

Existing repository scaling measurements for the native staging proof are approximately linear:

• 2,048 chunks: 497 ms
• 4,096 chunks: 1,021 ms
• 8,192 chunks: 2,150 ms
• 22,016 chunks: 5,704 ms
• approximately 243 to 259 microseconds per chunk

At 168,659 chunks, the digest proof alone projects to roughly 41 to 44 seconds, before sealed publication work. The fixed 30-second deadline therefore cannot admit this valid corpus.

Steps to reproduce

1. Run the PR feat: TraceDecay V2 delivery (PR8–PR13) + daemon performance, memory, and reliability overhaul #707 semantic recovery stack with a retained staged generation of roughly 168k chunks.
2. Allow the retained generation to resume and finish staging.
3. Drive final semantic publication.
4. Observe publish_generation terminate at the graph authority deadline and semantic runtime become degraded.

Error output

semantic publication phase: publish_generation
graph authority check: deadline exceeded
runtime state: degraded
runtime reason: runtime_failure
active generation: null

Expected behavior

Daemon-owned whole-generation verification and publication should be bounded by lifecycle cancellation and a background-operation budget appropriate to the existing repository policy. Interactive graph operations should retain their short deadline.

Each independently corpus-scaled publication phase should receive a fresh authority so one full-generation pass cannot consume the budget of the next pass.

Component

Indexing / sync

Additional context

Related to #792, but not fixed by it. #792 removes the stale begin_generation blocker; this defect appears later, once that recovery succeeds and the retained generation reaches final publication.

The retained stage remains available for replay. Reproduction does not require rebuilding embeddings or creating a second generation.

[ScriptedAlchemy]

Implementation is pushed on codex/issue-837-semantic-publication at f792c5512 (rebased onto #707 commit 39231da3f).

Root cause confirmed: publish_generation_records reused one absolute 30-second authority across prepare_publication_from_staged_native and publish_ready_stage, so the corpus-scaled prepare pass consumed the publish pass deadline.

The change keeps the 30-second authority for the interactive entry checkpoint, exposes the existing 168-hour/lifecycle-cancelled background graph budget from the usecases authority, and mints an independently fresh lifecycle-cancelled authority for each corpus-scaled phase.

Falsifiable RED/GREEN evidence:

• RED on the pre-fix code: exact test ran 1 and failed with typed DeadlineExceeded in 0.10s.
• GREEN after the fix and after rebase: transition tests 4 passed / 0 failed / 589 filtered.
• cargo check -p tracedecay-usecases --features hotpath: passed.
• cargo clippy -p tracedecay-usecases --all-targets -- -D warnings: passed.

The issue remains open intentionally. Final integration must replace the private duplicate GRAPH_BACKGROUND_OPERATION_BUDGET in tracedecay-store-runtime/src/session_registry/code_graph.rs with the new canonical usecases export, then rerun the combined verification. That file currently has active work in the shared integration checkout, so this branch does not overwrite it.

[ScriptedAlchemy]

Updated combined fix branch codex/issue-837-semantic-publication at 6daf3e839 (rebased on PR tip 298424812; #837 commit 7807f83e1). Interactive graph operations retain the 30s authority. Corpus-scaled prepare and publish each receive a fresh lifecycle-cancelled authority using the exported canonical usecases background budget; no limit was raised, disabled, or made configurable. Post-rebase evidence: hauler cc-3742 transition tests 4/4 (including the deterministic fresh-authority regression), cc-3744 hotpath feature check clean, and cc-3745 clippy for usecases + store-runtime all targets with -D warnings clean. Keeping the issue open: dirty session_registry/code_graph.rs was intentionally not touched and still declares the duplicate 168h downstream constant; integration must replace it with the exported usecases authority before closure.

[ScriptedAlchemy]

Review follow-up is pushed on codex/issue-837-semantic-publication at 4c393b09a, rebased on current PR tip 0aab74a74. The prior duplicate-budget integration caveat is now resolved by 4c393b09a: store-runtime imports the public usecases GRAPH_BACKGROUND_OPERATION_BUDGET and deletes its local 168h copy. #837 publication authority remains in 42ec3858a: interactive operations stay at 30s, while independently fresh corpus-phase authorities use the canonical background policy and runtime lifecycle cancellation. Post-rebase evidence: cc-3905 transition tests 4/4 (including fresh prepare/publish authority), cc-3906 store-runtime background-budget consumer tests 2/2, cc-3907 usecases + store-runtime all-target clippy with -D warnings, and workspace cargo fmt --all -- --check plus commitlint clean. Keeping open for integration/deployment verification.

[ScriptedAlchemy]

The reviewed root-cause fix has now landed on PR #707 at e92af9e. Integrated verification: restart/cancellation 1/1, transition suite 4/4, and sealed-projection deadline suite 2/2. This remains open for the required deployment/profile verification.

[ScriptedAlchemy]

The corpus-scaled publication phase/deadline and restart/cancellation fixes are central on PR #707, including commit 4d56f59. Keeping #837 open for the guarded real-profile publication across the former ~168k-unit boundary; no deadline was raised and focused tests are not a substitute for that deployment proof.

[ScriptedAlchemy]

Landed on codex/tracedecay-total-redesign-plan-reopened at 52b7869 (fix(store): give each publication pass its own background authority).

What changed: publish_generation_records ran the whole-generation digest proof and the install pass under one 30 s interactive GRAPH_OPERATION_DEADLINE authority, so at ~168k chunks the digest pass (~250 µs/chunk) spent the install pass's budget before it started and the runtime degraded to runtime_failure. Each corpus-scaled pass now mints its own fresh authority bounded by lifecycle cancellation and the background budget; load_published_generation_snapshot takes the caller's recovery budget instead of minting its own 30 s one (restart/open hydration no longer silently revert to the interactive deadline). Neither deadline moved and verification is unchanged. Each phase emits a typed VectorPublicationPhaseV1 observation with chunk count and elapsed time, so the minutes of previously silent work are visible.

Verification (own target dir, plain cargo): clippy -D warnings on tracedecay-usecases / tracedecay-store-runtime / tracedecay all-targets clean; tracedecay-usecases --lib semantic_runtime store::vector_generations 144 passed; tracedecay-store-runtime --lib code_graph 33 passed, 1 ignored.

Not yet measured at corpus scale on the shipped path; leaving open until a large-corpus run shows the phases complete under the background authority. The per-phase timings from #838's attribution run (410b4db) are the input for that.