processor: export the final partial batch on shutdown - #874
Open
damilolaedwards wants to merge 1 commit into
Open
processor: export the final partial batch on shutdown#874damilolaedwards wants to merge 1 commit into
damilolaedwards wants to merge 1 commit into
Conversation
The batch builder holds accumulated items in an in-memory batch until they reach the export size or the flush timer fires. On shutdown the builder ran in a goroutine that was never waited on, and drainQueue only waited on the queue and the worker channel, not on that in-memory batch. As a result close(stopWorkersCh) raced the builder: the builder could exit without flushing, or its final batch could sit in the worker channel while the workers stopped. Any batch smaller than the export size, which is the common case on a graceful restart, was silently dropped with no failure recorded. Track the batch builder with its own wait group and reorder shutdown: close the queue, wait for the builder to flush its final batch, wait for the workers to pick up every handed-off batch, and only then stop the workers. In-flight exports still complete before Shutdown returns.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The batch builder holds accumulated items in an in-memory batch until they reach the export size or the flush timer fires. On shutdown the builder ran in a goroutine that was never waited on, and drainQueue only waited on the queue and the worker channel, not on that in-memory batch.
As a result close(stopWorkersCh) raced the builder: the builder could exit without flushing, or its final batch could sit in the worker channel while the workers stopped. Any batch smaller than the export size, which is the common case on a graceful restart, was silently dropped with no failure recorded.
This tracks the batch builder with its own wait group and reorders shutdown: close the queue, wait for the builder to flush its final batch, wait for the workers to pick up every handed-off batch, and only then stop the workers. In-flight exports still complete before Shutdown returns.
Tests: adds a shutdown test over one and four workers asserting the final partial batch is always exported. Ran the full processor suite repeatedly under the race detector.