fix: confirm durable writes before advancing state in pruning and system-contract indexers - #816
Open
damilolaedwards wants to merge 1 commit into
Conversation
…tem-contract indexers Two more instances of the same bug class fixed in ethpandaops#784 (tx matcher, bid cache, finalizeEpoch), missed because they live in different files. processEpochPruning discarded RunDBTransaction's return value entirely (not even logged) and always returned a nil error, so indexer.lastPrunedEpoch advanced and the pruned blocks' bodies/epoch stats were evicted from cache regardless of whether the persist transaction - which durably records the prune checkpoint - actually committed. A transient write failure on the default SQLite engine silently and permanently lost that epoch's data until a process restart, with no error surfaced anywhere. persistFinalizedRequestTxs/persistRecentRequestTxs in the system-contract event indexers (deposit/withdrawal/consolidation/builder-deposit/builder-exit) mutated the in-memory FinalBlock/ForkStates cursor inside the RunDBTransaction closure before persistState's commit was confirmed, with no rollback on failure - so a failed persist permanently skipped that block range for the process's uptime. persistState itself had the same problem one level down: it deletes stale ForkStates entries before attempting the write, also without rolling back on failure. Fix: capture the transaction error in all cases and roll back exactly what was advanced/removed if it's non-nil, so the range is retried on the next cycle instead of silently disappearing. Tests: each fix has a test that fails without it and passes with it, verified by temporarily reverting the fix and confirming the new tests fail as expected, then restoring it. go build, go vet, gofmt, and the full test suite (including -race) are clean.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two more instances of the bug class fixed in #784 (tx matcher, bid cache,
finalizeEpoch), missed there because they live in different files.Beacon epoch pruning silently loses data on a failed write
processEpochPruningcalleddb.RunDBTransaction(...)as a bare statement, the returned error was never captured, not even logged.indexer.lastPrunedEpochthen advanced unconditionally, and the pruned blocks' bodies/epoch stats were evicted from cache regardless of whether the transaction (which durably records the prune checkpoint viaupdatePruningState) actually committed. The function's own error return was alwaysnil, so the caller (runCachePruning) had no way to detect or retry the failure either. This runs on every finalization-driven pruning cycle, i.e. constantly, on the default (SQLite) config.Fix: capture the transaction error; on failure, return before advancing
lastPrunedEpochor touching the cache, so the epoch is retried on the next pruning cycle instead of silently disappearing.System-contract indexer cursor advances before its write is confirmed
persistFinalizedRequestTxs/persistRecentRequestTxs(deposit/withdrawal/consolidation/builder-deposit/builder-exit indexers) mutated the in-memoryFinalBlock/ForkStatescursor inside theRunDBTransactionclosure beforepersistState's commit was confirmed, with no rollback on failure.ci.stateis loaded once at process start and drives every subsequent block-range computation, so a failed persist permanently skips that range for the rest of the process's uptime.persistStateitself had the same problem one level down: it deletes staleForkStatesentries before attempting the write, also without rolling back on failure.Fix: both callers now snapshot the value they're about to set and roll back to it if the transaction fails;
persistStaterestores exactly what it removed if its write fails.Tests
Each fix has a test that fails without it and passes with it, verified by temporarily reverting the fix, confirming the new tests fail as expected, then restoring it:
pruning_test.go: failed persist doesn't advancelastPrunedEpoch; successful persist still does.contract_indexer_durable_write_test.go: failed persist rolls backFinalBlock/FinalQueueLen, rolls back a newly-setForkStatesentry to absent, restores a pre-existingForkStatesentry to its prior value, andpersistState's own internal cleanup restores fork states it removed.go build,go vet,gofmt, and the full test suite (including-race) are clean.