Skip to content

fix: confirm durable writes before advancing state in pruning and system-contract indexers - #816

Open
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/pruning-and-contract-indexer-state-advance
Open

fix: confirm durable writes before advancing state in pruning and system-contract indexers#816
damilolaedwards wants to merge 1 commit into
ethpandaops:masterfrom
damilolaedwards:fix/pruning-and-contract-indexer-state-advance

Conversation

@damilolaedwards

@damilolaedwards damilolaedwards commented Aug 4, 2026

Copy link
Copy Markdown
Contributor

Summary

Two more instances of the bug class fixed in #784 (tx matcher, bid cache, finalizeEpoch), missed there because they live in different files.

Beacon epoch pruning silently loses data on a failed write

processEpochPruning called db.RunDBTransaction(...) as a bare statement, the returned error was never captured, not even logged. indexer.lastPrunedEpoch then advanced unconditionally, and the pruned blocks' bodies/epoch stats were evicted from cache regardless of whether the transaction (which durably records the prune checkpoint via updatePruningState) actually committed. The function's own error return was always nil, so the caller (runCachePruning) had no way to detect or retry the failure either. This runs on every finalization-driven pruning cycle, i.e. constantly, on the default (SQLite) config.

Fix: capture the transaction error; on failure, return before advancing lastPrunedEpoch or touching the cache, so the epoch is retried on the next pruning cycle instead of silently disappearing.

System-contract indexer cursor advances before its write is confirmed

persistFinalizedRequestTxs/persistRecentRequestTxs (deposit/withdrawal/consolidation/builder-deposit/builder-exit indexers) mutated the in-memory FinalBlock/ForkStates cursor inside the RunDBTransaction closure before persistState's commit was confirmed, with no rollback on failure. ci.state is loaded once at process start and drives every subsequent block-range computation, so a failed persist permanently skips that range for the rest of the process's uptime. persistState itself had the same problem one level down: it deletes stale ForkStates entries before attempting the write, also without rolling back on failure.

Fix: both callers now snapshot the value they're about to set and roll back to it if the transaction fails; persistState restores exactly what it removed if its write fails.

Tests

Each fix has a test that fails without it and passes with it, verified by temporarily reverting the fix, confirming the new tests fail as expected, then restoring it:

  • pruning_test.go: failed persist doesn't advance lastPrunedEpoch; successful persist still does.
  • contract_indexer_durable_write_test.go: failed persist rolls back FinalBlock/FinalQueueLen, rolls back a newly-set ForkStates entry to absent, restores a pre-existing ForkStates entry to its prior value, and persistState's own internal cleanup restores fork states it removed.

go build, go vet, gofmt, and the full test suite (including -race) are clean.

…tem-contract indexers

Two more instances of the same bug class fixed in ethpandaops#784 (tx matcher, bid cache,
finalizeEpoch), missed because they live in different files.

processEpochPruning discarded RunDBTransaction's return value entirely (not even
logged) and always returned a nil error, so indexer.lastPrunedEpoch advanced and
the pruned blocks' bodies/epoch stats were evicted from cache regardless of
whether the persist transaction - which durably records the prune checkpoint -
actually committed. A transient write failure on the default SQLite engine
silently and permanently lost that epoch's data until a process restart, with no
error surfaced anywhere.

persistFinalizedRequestTxs/persistRecentRequestTxs in the system-contract event
indexers (deposit/withdrawal/consolidation/builder-deposit/builder-exit) mutated
the in-memory FinalBlock/ForkStates cursor inside the RunDBTransaction closure
before persistState's commit was confirmed, with no rollback on failure - so a
failed persist permanently skipped that block range for the process's uptime.
persistState itself had the same problem one level down: it deletes stale
ForkStates entries before attempting the write, also without rolling back on
failure.

Fix: capture the transaction error in all cases and roll back exactly what was
advanced/removed if it's non-nil, so the range is retried on the next cycle
instead of silently disappearing.

Tests: each fix has a test that fails without it and passes with it, verified by
temporarily reverting the fix and confirming the new tests fail as expected, then
restoring it. go build, go vet, gofmt, and the full test suite (including -race)
are clean.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant