Repository navigation
[EPIC] Better support / documentation / tests for Streaming SQL Engines #24265
Description
Activity
- changed the title
[-][EPIC] Better support for Streaming SQL[/-][+][EPIC] Better support / documentation / tests for Streaming SQL[/+]on Aug 11, 2026 - changed the title
[-][EPIC] Better support / documentation / tests for Streaming SQL[/-][+][EPIC] Better support / documentation / tests for Streaming SQL Engines[/+]on Aug 11, 2026 Thanks for creating this issue, @alamb!
I think getting something like "Size-or-deadline" flush trigger into as many operators as possible (with values configured globally), can be the first step into better streaming semantics. You can avoid a global "streaming" flag by simply choosing how to flush batches, for example:
- Batch system: size: 8k, deadline: disabled.
- Streaming system: size: 1k, deadline: 100ms.
Control-message propagation also would be a really cool building block: you could build high-level functionality like checkpointing, windowing, etc.
Reacted by jling, Andrew Lamb, SSpirits and Aldrin MThanks @alamb !
In my personal experience building out StreamFusion, DataFusion could cover a few more streaming join operations if it supported a couple of things:
- Changelogs (need to be able to support incoming deleted rows, updated rows, etc)
- Window functions having an API to take in existing state and then aggregate an incoming batch into them
@sap1ens - as StreamFusion and Streamling continue to both evolve a bit more, I'd love to hop on a call sometime and chat about them!
Reacted by Andrew Lamb and Jonathan ChenThanks for this @alamb.
Massive also has an internal stream processing engine built on DataFusion. This talk from the Stockholm meetup earlier this year provides an overview.
We ultimately opted out of DataFusion's physical layer and built our own for a few reasons:
- Our implementation requires support for interleaving non-data payloads with data across operator edges. The core algorithm we use to provide exactly-once processing guarantees relies on these payloads to establish ordered cuts within the streaming graph.
- Edges between operators can have different semantics, such as append-only and changelog streams.
- Streaming engines also have specific requirements around watermark alignment, stateful node IDs, and stream keys.
It would be great if DataFusion's physical execution/planning interfaces could accommodate some of these requirements, though a generic extension mechanism may be a better fit than baking stream-processing-specific semantics into DataFusion itself.
Separately, we do a fair amount with aggregation and another pain point I've come across is the lack of a way to peek at aggregate state or evaluate only a subset of a group. Supporting this would make incremental emission from time-windowed aggregation operators more efficient. This issue describes a similar problem.
I'm sure there is more that I'm not thinking of right now 😅 . Will update this thread as more comes to mind.
Reacted by Andrew Lamb, Matthew Turner and Max MeldrumI think https://github.com/feldera/feldera is very interesting in this space
Reacted by Alessandro Solimando, Andrew Lamb and SSpiritsHere is a follows up discussion for
My suggestion is to completely separate the streaming variant from the existing implementation—using different Streams/state machines—instead of adding internal control flags and reusing the same implementation.
The reason is that adding a new flag for "streaming vs. batch" does not just add a small amount of complexity. It adds a new dimension to the operator's state space, and the complexity grows combinatorially as more such dimensions are introduced.
This is somewhat counterintuitive. What #24353 is doing may look like a small, linear increase in implementation complexity, but it is actually adding another independent dimension to the operator, and the extra complexity will continue to grow on that newly added dimension.
There are already other dimensions, such as ordered vs. unordered input, spilling vs. non-spilling, etc. 10 innocent-looking binary dimensions can already produce 1,024 combinations. We have suffered from the same problem before; the following write-up is a case study:
The alternative I propose is to represent these as hard separated stream types, such as
StreamingFilterStreamandFilterStream, each with its own state machine.EDIT: I suspect some optimizer rules are not compatible with streaming variants for the existing
ExecutionPlans, it might be necessary to even split theExecutionPlanand maintain the dedicated optimizer rules for streaming input. Then a good long-term approach looks like a) only keep API changes indatafusioncore b) move implementation to a separate project.Reacted by Andrew Lamb and Adrian Garcia BadaraccoI agree on separating into its own Stream type. If we find significant similarity between the implementations once the streaming feature is in place, we can revisit and refactor at that point.
Separately, we do a fair amount with aggregation and another pain point I've come across is the lack of a way to peek at aggregate state or evaluate only a subset of a group. Supporting this would make incremental emission from time-windowed aggregation operators more efficient.
I filed two related issues with concrete API proposals:
- Add selective, non-destructive reads for grouped aggregation state #24602
- Add arbitrary group retention for grouped aggregation state #24603
The motivating use case is stateful stream processing, but both APIs are general and do not necessarily tie DataFusion to any sort of stream processing model.
Pulling some discussion out into #24353 (comment)
If we agree on supporting more "streaming" execution in DataFusion, I think we should extend that support across all existing operators. +1 from me on the overall direction. How to design this while keeping maintenance complexity low could be a separate discussion — maybe in #24265
In my mind DataFusion is in a half way state now -- it has some features / support for streaming (e.g.
Boundedness), but as shown in #24044 since there is no clear design / tests / documentation implicit assumptions that streaming systems may be counting on can (and do) get brokenSo I think we should first agree on if we want to try and make DataFusion more useful for building streaming engines -- I think it is important and there are a bunch of people already doing so (and have done so for a while -- e.g. Arroyo and earlier versions of Synnada). Rewriting Apache Flink in rust seems to be all the rage now too (e.g. @jordepic 's StreamFusion, etc) , and many people are using DataFusion to try.
it seems like that is the consensus of this ticket too
Given @2010YOUY01 's comments on #24265 (comment) and @jayzhan211 's comments
I agree on separating into its own Stream type. If we find significant similarity between the implementations once the streaming feature is in place, we can revisit and refactor at that point.
Where do we want to put this code?
It feels to me like we should try and put it into its own crate (where we can document it, and allow downstream systems to opt in) -- BTW the symmetric hash join code is another example of something that I think is mostly for the streaming usecase and would be good to consolidate
How about a new crate
datafusion-streaming?
Reacted by Bhargava Vadlamani and Yongting You
Is your feature request related to a problem or challenge?
DataFusion has traditionally been developed using techniques from Batch processing systems and tries to maximize throughput by being as efficient as possible per tuple. This often involves techniques such as batching rows together to amortize the overhead of each row
However, it also seems increasingly common to try and use DataFusion for streaming systems, similar to Apache Flink, which typically have "unbounded queries" that run indefinitely and place a high value on values being produced quickly (to minimize latency) rather than simply raw per-row computation.
I think we have a bunch of features for streaming documented / tested in isolation already (see below) but the overall design and goal is not clearly documented or tested and thus the overall experience is regressing
The recent presentation on youtube from @sap1ens also mentioned this mismatch between streaming and batch oriented workflows for several built in operators:
It turns out that unfortunately we are making it worse for streaming engines recently by making things better for batch execution (typically by moving buffering into the execution nodes)
RepartitionExecRepartitionExecwithholds all output from an unbounded input untilbatch_sizerows accumulate #24044FilterExec(DF 52), Remove FilterExec from CoalesceBatches optimization rule, add fetch support #18630 / Remove coalesce batches rule and deprecate CoalesceBatchesExec #19622 removedCoalesceBatchesentirely so buffering is now inside the operatorThere is some basic infrastructure today, but it isn't used / respected everywhere and there isn't an over arching strategy / description of its use
However, given that we are regressing behavior for streaming I think we should consider formalizing this functionality more
Example Streaming Systems
Describe the solution you'd like
Given how many systems seem to want to use DataFusion for streaming systems (rather than batch oriented ones) and the natural tension between batching (maximize throughput) vs streaming (minimize latency) I think if we should formalize / test / document the best way to configure DataFusion for streaming.
Describe alternatives you've considered
Better documentation / tests
Maybe part of this epic could be just to document what we have better
UNBOUNDEDtables) #9016Config Flag
Maybe it would be worth some sort of "bounded configuration mode" setting to make implementing streaming systems easier
For example,
streamingoroptimize_for_latency(maybe repurpose datafusion.execution.coalesce_batches (source) -- since Remove coalesce batches rule and deprecate CoalesceBatchesExec #19622 I don't think anything reads itBetter propagation / policies
Several potential policies for when operators should flush partially-full batches came up in the context of #24044:
fix: avoid buffering unbounded repartition output indefinitely #24193 for
RepartitionExec): simple and fixes thecorrectness issue, but @calvinchengx (comment) and @jayzhan211 (comment) note it may not be applicable to all cases
Poll::Ready, but flush theresidual whenever the input returns
Pendinginstead of holding it.batch_sizeOR after aconfigurable maximum batch age, whichever comes first.
watermark / checkpoint-barrier-based, if DataFusion later grows control-message propagation
Make more operators respect buffering
Additional context
We have discussed this in the past
execution_modewithemission_typeandboundedness#13823Related Issues
UNBOUNDEDtables) #9016RepartitionExecwithholds all output from an unbounded input untilbatch_sizerows accumulate #24044 (fix proposed in fix: avoid buffering unbounded repartition output indefinitely #24193)AccumulatorsandScalarValueserializable #11369