server: release beacon block dedup mark when a sink write fails - #871
Open
damilolaedwards wants to merge 1 commit into
Open
Conversation
Beacon block events are deduplicated by marking their state root in a shared cache during filtering. The mark was set before the events reached a sink, and it was never released if the sink write failed. When a sink was unavailable the client retried the batch, but the retry found the mark already set and dropped the block as a duplicate, so it was lost for the cache TTL even though it was never durably written. Record the key marked during filtering, return it from the handler, and release it if any sink write fails so a retry is processed normally. Duplicate handling on the success path is unchanged.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Beacon block events are deduplicated by marking their state root in a shared cache during filtering. The mark was set before the events reached a sink, and it was never released if the sink write failed.
When a sink was unavailable the client retried the batch, but the retry found the mark already set and dropped the block as a duplicate. The block was therefore lost for the cache TTL despite never being durably written. Because the key is shared across sentries, every sentry reporting the same block during the outage window was dropped the same way.
This records the key marked during filtering, returns it from the handler, and releases it if any sink write fails, so a retry is processed normally. Duplicate handling on the success path is unchanged, and the only added cost is a re-processed batch on the failure path whose duplicates are absorbed by the ReplacingMergeTree tables downstream.
Scope is limited to the two event types that deduplicate against the cache via a new optional interface, so the other event handlers are untouched.
Tests: adds pipeline coverage for releasing the mark on sink failure so a retry is not dropped, plus a control confirming duplicates are still dropped after a successful write.