RepartitionExec withholds all output from an unbounded input until batch_size rows accumulate

Describe the bug

RepartitionExec coalesces on the producer side. OutputChannel::coalesce
pushes each batch into a LimitedBatchCoalescer and forwards only batches that
have reached target_batch_size (SessionConfig::batch_size(), default 8192).
Whatever is left over is flushed by SharedCoalescer::finalize, which runs when
the last input sender finishes.

An unbounded input never finishes, so the residual is never flushed. A stream
that produces rows slowly, or in small batches, delivers nothing downstream
until 8192 rows have accumulated — however long that takes. The plan is not
deadlocked and no data is lost: it is buffered, invisibly, for an unbounded
amount of time.

The coalescer is created for every non-preserve-order repartition, without
consulting the input's boundedness:

https://github.com/apache/datafusion/blob/main/datafusion/physical-plan/src/repartition/mod.rs#L528

let shared_coalescer = (!preserve_order).then(|| {
    SharedCoalescer::new(
        input.schema(),
        context.session_config().batch_size(),
        num_input_partitions,
    )
});

ExecutionPlan::boundedness() reports Boundedness::Unbounded for such an
input, but is not consulted here.

To Reproduce

Full program, depending only on datafusion = "54.1.0", futures, and tokio.
A PartitionStream emits one row every 100 ms and never ends; it is fed through
RepartitionExec and the output is drained for six seconds.

src/main.rs

//! `RepartitionExec` withholds all output from an unbounded input until
//! `batch_size` rows accumulate.
//!
//! A source emits one row every 100 ms and never ends. It is fed through
//! `RepartitionExec` with round-robin partitioning, and the output is drained
//! for six seconds. About 60 rows are produced in that window.
//!
//! Run with `cargo run --release`.

use std::sync::Arc;
use std::time::{Duration, Instant};

use datafusion::arrow::array::{Int64Array, RecordBatch};
use datafusion::arrow::datatypes::{DataType, Field, Schema, SchemaRef};
use datafusion::common::Result;
use datafusion::execution::{SendableRecordBatchStream, TaskContext};
use datafusion::physical_expr::Partitioning;
use datafusion::physical_plan::repartition::RepartitionExec;
use datafusion::physical_plan::stream::RecordBatchStreamAdapter;
use datafusion::physical_plan::streaming::{PartitionStream, StreamingTableExec};
use datafusion::physical_plan::{ExecutionPlan, execute_stream};
use datafusion::physical_expr::LexOrdering;
use datafusion::prelude::{SessionConfig, SessionContext};
use futures::StreamExt;

/// One row every 100 ms, forever.
#[derive(Debug)]
struct TickStream {
    schema: SchemaRef,
}

impl PartitionStream for TickStream {
    fn schema(&self) -> &SchemaRef {
        &self.schema
    }

    fn execute(&self, _ctx: Arc<TaskContext>) -> SendableRecordBatchStream {
        let schema = Arc::clone(&self.schema);
        let stream = futures::stream::unfold(0i64, move |i| {
            let schema = Arc::clone(&schema);
            async move {
                tokio::time::sleep(Duration::from_millis(100)).await;
                let batch = RecordBatch::try_new(
                    Arc::clone(&schema),
                    vec![Arc::new(Int64Array::from(vec![i]))],
                );
                Some((batch.map_err(Into::into), i + 1))
            }
        });
        Box::pin(RecordBatchStreamAdapter::new(
            Arc::clone(&self.schema),
            stream,
        ))
    }
}

/// Drains the plan for `secs` and returns how many rows arrived.
async fn rows_delivered(batch_size: usize, secs: u64, repartition: bool) -> Result<usize> {
    let schema = Arc::new(Schema::new(vec![Field::new("v", DataType::Int64, false)]));

    let source = Arc::new(StreamingTableExec::try_new(
        Arc::clone(&schema),
        vec![Arc::new(TickStream {
            schema: Arc::clone(&schema),
        }) as Arc<dyn PartitionStream>],
        None,
        std::iter::empty::<LexOrdering>(),
        true, // infinite
        None,
    )?) as Arc<dyn ExecutionPlan>;

    let plan: Arc<dyn ExecutionPlan> = if repartition {
        Arc::new(RepartitionExec::try_new(
            source,
            Partitioning::RoundRobinBatch(4),
        )?)
    } else {
        source
    };

    let ctx = SessionContext::new_with_config(SessionConfig::new().with_batch_size(batch_size));
    let mut stream = execute_stream(plan, ctx.task_ctx())?;

    let deadline = Instant::now() + Duration::from_secs(secs);
    let mut rows = 0usize;
    while Instant::now() < deadline {
        match tokio::time::timeout(Duration::from_millis(250), stream.next()).await {
            Ok(Some(batch)) => rows += batch?.num_rows(),
            Ok(None) => break,
            Err(_) => {} // no batch this interval; keep waiting
        }
    }
    Ok(rows)
}

#[tokio::main]
async fn main() -> Result<()> {
    let secs = 6;
    println!("source emits 1 row / 100 ms, so ~{} rows per run\n", secs * 10);

    let baseline = rows_delivered(8192, secs, false).await?;
    println!("no repartition,   batch_size = 8192 -> {baseline} rows");

    let repart_default = rows_delivered(8192, secs, true).await?;
    println!("RoundRobinBatch(4), batch_size = 8192 -> {repart_default} rows");

    let repart_small = rows_delivered(1, secs, true).await?;
    println!("RoundRobinBatch(4), batch_size =    1 -> {repart_small} rows");

    Ok(())
}

Output of cargo run --release:

source emits 1 row / 100 ms, so ~60 rows per run

no repartition,   batch_size = 8192 -> 59 rows
RoundRobinBatch(4), batch_size = 8192 -> 0 rows
RoundRobinBatch(4), batch_size =    1 -> 59 rows

The middle line is the bug: the same source, the same six seconds, one
RepartitionExec in between, and nothing arrives. The first line shows the
source itself is fine, and the third isolates the coalescer — the repartition is
still there, only batch_size changed.

Reproduced on 54.1.0; the construction on main is unchanged.

Expected behavior

Rows reach the consumer at a bounded latency, as they do when the same plan runs
without a repartition, or with a small batch_size.

For an unbounded input, waiting for a full batch is not a bounded wait. A few
directions, in the order I would guess you prefer them — though this is your
design call, which is why this is an issue and not a pull request:

Skip the producer-side coalescer when the input is Boundedness::Unbounded.
Keep it, but flush on a time bound as well as a size bound.
Expose it as a config so an unbounded pipeline can turn it off.

Additional context

Some notes to save duplicated work:

This is not a regression from Integrate BatchCoalescer into RepartitionExec #18782. The producer-side coalescer arrived
with that issue, which also removed the coalesce_batches optimizer rule that
previously inserted a CoalesceBatchesExec after RepartitionExec. I checked
the older rule (52.0.0) and it did not consider boundedness either, so the
blind spot predates the integration — Integrate BatchCoalescer into RepartitionExec #18782 moved it inside the operator
rather than introducing it.
FilterExec looks like it has the same shape — it holds a
LimitedBatchCoalescer and I did not see a boundedness check near it. I have
not verified that one behaves the same way, so treat it as a lead rather
than a second report.
Found while debugging a streaming engine built on DataFusion, where a
round-robin repartition sat between an unbounded source and a sink and the
query produced no output at all. Worked around downstream by not repartitioning
streaming plans; the underlying behaviour is unchanged, which is why I am
reporting it here.

I think we need emission rules beyond a size-based trigger for unbounded streams. Possible strategies include:

Time-based: Emit if no new input arrives within a configurable time interval.
Count-based: Emit after N consecutive polls that return no input, indicating the stream is idle.
Watermark/checkpoint barrier-based: Emit when a watermark or checkpoint barrier indicates that all relevant events have been observed, following event-time semantics.

@calvinchengx

take

@calvinchengx I plan to address this in RepartitionExec by skipping producer-side batch coalescing when the input is unbounded, while preserving the current behavior for bounded inputs.

If a time- or idle-based flush is preferred, I can adjust the approach before opening the PR. Let me know. thanks!

Thanks for picking this up.

Your approach sounds right to me as a first step. If you're willing, a time or idle bound would be better still — skipping coalescing entirely just relocates the small batches downstream, and anything that commits or writes per batch then sees a lot of very small ones.

Either way, happy to test the PR.

I am looking for things we should include in the 55.0.0 release (#22393) and I found this one

It isn't 100% lear to me if this is a bug or if it is a change in behavior. Do you think we should hold the release for the resolution of this issue?

@alamb

@alamb I believe this is a bug because an incremental unbounded input can have available rows withheld indefinitely.

I opened the #24193 for this issue with the scoped fix discussed here. Can you please review and let me know if any suggestions. Thanks
cc @calvinchengx

take

If we disable coalescing for unbounded stream, then we might encounter high latency for many small batches. I think in the long term we still need mechanism used for streaming. But I agree disabling coalescing could be the first step to solve never emitting issue

@alamb

@alamb It's a bug rather than a change in behaviour. I don't think it needs to hold 55.0.0, though that is your call.

A bug because rows that are already available can be withheld indefinitely: the producer-side coalescer flushes its residual only when the input ends, and an unbounded input never does. For a source emitting small batches the practical effect is no output at all rather than delayed output. In the repro above, at the default 8192 and a source producing 10 rows/second, the first batch would appear after roughly 14 minutes.

Not a regression in this cycle, though. I checked 52.0.0: the coalesce_batches optimizer rule there had no boundedness handling either, so the external CoalesceBatchesExec it inserted after RepartitionExec had the same blind spot. #18782 moved coalescing inside the operator, which made a longstanding gap structural rather than introducing it. So holding the release would not be protecting anyone from something newly broken.

@jayzhan211

#24193 is the short-term correctness fix. unbounded repartitions bypass the completion-driven SharedCoalescer, ensuring partial batches are emitted while bounded inputs retain the existing size-based coalescing.

@jayzhan211 as per your comment too, i feel longer term, restoring unbounded coalescing with a size-or-deadline policy per output partition—flush at batch_size, a configurable maximum batch age, or completion.
A real timer provides deterministic progress; poll-count is scheduler-dependent, while watermark/barrier flushing can remain an optional extension if DataFusion later introduces control-message propagation would help. Tx!

Size-or-deadline sounds right to me, and I agree a real timer beats poll counting: poll frequency tells you about the scheduler rather than about the data.

One number in support of the deadline half, since it is easy to underweight. I measured a file sink that commits once per input batch, fed by a source emitting 1 to 2 rows per batch: 7,667 files for 7,667 rows in 8 seconds, about 958 files per second. That is what no coalescing costs a couple of operators downstream. Correct, but not something you would point at object storage.

The other thing worth settling early is the default maximum batch age, not just the knob. If the default is long, a slow stream still looks stalled and it is this same report with a smaller constant.

size-or-deadline policy

I think there is a more sensible default without new deadlines if the user doesn't specify an explicit intent. Namely, bias unbounded execution towards low latency. What if to drain all Poll::Ready batches, i.e. until there is an await point?

This might achieve low-latency and optimal performance more likely by default (batches are hot). I assume, pure compute nodes generally don't insert Pending. If so, one could go look for optimizations as long as they don't play directly against cooperative scheduling.

FilterExec looks like it has the same shape — it holds a LimitedBatchCoalescer and I did not see a boundedness check near it.

datafusion/datafusion/physical-plan/src/filter.rs

Lines 1256 to 1316 in b225ded

    
                   // Attempt to pull the next batch from the input stream. 
        
                   match ready!(self.input.poll_next_unpin(cx)) { 
        
                       None => { 
        
                           self.batch_coalescer.finish()?; 
        
                           // Release the input pipeline's resources. 
        
                           let input_schema = self.input.schema(); 
        
                           self.input = Box::pin(EmptyRecordBatchStream::new(input_schema)); 
        
                           // continue draining the coalescer 
        
                       } 
        
                       Some(Ok(batch)) => { 
        
                           let timer = elapsed_compute.timer(); 
        
                           let status = self.predicate.as_ref() 
        
                               .evaluate(&batch) 
        
                               .and_then(|v| v.into_array(batch.num_rows())) 
        
                               .and_then(|array| { 
        
                                   Ok(match self.projection.as_ref()  { 
        
                                       Some(projection) => { 
        
                                           let projected_batch = batch.project(projection)?; 
        
                                           (array, projected_batch) 
        
                                       }, 
        
                                       None => (array, batch) 
        
                                   }) 
        
                               }).and_then(|(array, batch)| { 
        
                                   match as_boolean_array(&array) { 
        
                                       Ok(filter_array) => { 
        
                                           self.metrics.selectivity.add_total(batch.num_rows()); 
        
                                           // TODO: support push_batch_with_filter in LimitedBatchCoalescer 
        
                                           let batch = filter_record_batch(&batch, filter_array)?; 
        
                                           let state = self.batch_coalescer.push_batch(batch)?; 
        
                                           Ok(state) 
        
                                       } 
        
                                       Err(_) => { 
        
                                           internal_err!( 
        
                                               "Cannot create filter_array from non-boolean predicates" 
        
                                           ) 
        
                                       } 
        
                                   } 
        
                               })?; 
        
                           timer.done(); 
        
                           match status { 
        
                               PushBatchStatus::Continue => { 
        
                                   // Keep pushing more batches 
        
                               } 
        
                               PushBatchStatus::LimitReached => { 
        
                                   // limit was reached, so stop early 
        
                                   self.batch_coalescer.finish()?; 
        
                                   // Release the input pipeline's resources. 
        
                                   let input_schema = self.input.schema(); 
        
                                   self.input = 
        
                                       Box::pin(EmptyRecordBatchStream::new(input_schema)); 
        
                                   // continue draining the coalescer 
        
                               } 
        
                           } 
        
                       } 
        
                       // Error case 
        
                       other => return Poll::Ready(other), 
        
                   } 
        
               } 
        
           }

I think FilterExec does have the same unfortunate behavior. To fix unbounded behavior it could more look like this:

  match self.input.poll_next_unpin(cx) {
      Poll::Pending if !self.batch_coalescer.is_empty() => {
          self.batch_coalescer.flush_buffered_batch()?; // drain but don't finish
          continue;
      }
      Poll::Pending => return Poll::Pending,
      Poll::Ready(None) => {
          self.batch_coalescer.finish()?;
          // ...
      }
      Poll::Ready(Some(batch)) => {
          // ...
      }
  }

Non-flushing of unbounded streams is not new though. It was introduced in #18604 and DataFusion 52. Before, it returned each non-empty filtered batch immediately. AI researched details:

Details

   When            Change                                            Effect
  ━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━  ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━
   Dec 30, 2020    Original CoalesceBatchesExec commit (https://     The optimizer wrapped FilterExec in a
                   github.com/apache/datafusion/                     coalescer. It flushed at target size or EOS,
                   commit/13dbe8ad9b958ad20c07602dc9b8d5ef3eb37c7    but returned Pending without flushing.
                   1), released in 4.0.0
  ──────────────  ────────────────────────────────────────────────  ─────────────────────────────────────────────────
   Dec 28, 2022    PR #4694 (https://github.com/apache/              Added official infinite-input support.
                   datafusion/pull/4694), released in 16.0.0         CoalesceBatchesExec declared itself compatible
                                                                     with unbounded input merely by propagating
                                                                     unbounded_output, without addressing residual
                                                                     flushing.
  ──────────────  ────────────────────────────────────────────────  ─────────────────────────────────────────────────
   Nov 11, 2025    PR #18604 (https://github.com/apache/             Moved the coalescer inside FilterExec, making
                   datafusion/pull/18604), released in 52.0.0        the behavior inherent to the operator.
  ──────────────  ────────────────────────────────────────────────  ─────────────────────────────────────────────────
   Nov 14, 2025    PR #18630 (https://github.com/apache/             Removed the now-redundant optimizer wrapper and
                   datafusion/pull/18630)                            added the current LimitedBatchCoalescer; the
                                                                     non-flushing behavior remained.

@sap1ens

A bug because rows that are already available can be withheld indefinitely: the producer-side coalescer flushes its residual only when the input ends, and an unbounded input never does. For a source emitting small batches the practical effect is no output at all rather than delayed output. In the repro above, at the default 8192 and a source producing 10 rows/second, the first batch would appear after roughly 14 minutes.

The counter to this is that for batch workloads, emitting more frequently is less efficient (as there is now more overhead per batch)

I recently watched a presentation from @sap1ens and he also mentioned this mismatch between streaming and batch oriented workflows for several built in operators: https://www.youtube.com/watch?v=0-BIHyzODH8&t=2s

Given how many systems seem to want to use DataFusion for streaming systems (rather than batch oriented ones) and the natural tension between batching (maximize throughput) vs streaming (minimize latency) I wonder if we should try and add some way to configure which behavior is desired.

For example, there is a way for operators today to report their boundedness behavior: https://docs.rs/datafusion/latest/datafusion/physical_plan/trait.ExecutionPlanProperties.html#tymethod.boundedness

Maybe it would be worth some sort of "bounded configuration mode" setting to make implementing streaming systems easier

For example,

A global config setting like streaming or optimize_for_latency
A flag on operators like FilterExec and RepartitionExec that controls their emission behavior (aka should they flush at the earliest opportunity)

I think there is a more sensible default without new deadlines if the user doesn't specify an explicit intent. Namely, bias unbounded execution towards low latency. What if to drain all Poll::Ready batches, i.e. until there is an await point?

@ahirner, this is an interesting idea to try and avoid the flag and figure it out from the input declarations.

to be clear, I think we should consider this (make it easier to build streaming systems on DataFusion as a follow on task / not hold DataFusion 55 for it, nor really treat it like a bug). I will file a ticket to track the idea

Filed a ticket to try and orgnaize this better

[EPIC] Better support / documentation / tests for Streaming SQL Engines #24265

fix: avoid buffering unbounded repartition output indefinitely (apach…

Navigation Menu

Repository navigation

`RepartitionExec` withholds all output from an unbounded input until `batch_size` rows accumulate #24044

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

	// Attempt to pull the next batch from the input stream.
	match ready!(self.input.poll_next_unpin(cx)) {
	None => {
	self.batch_coalescer.finish()?;
	// Release the input pipeline's resources.
	let input_schema = self.input.schema();
	self.input = Box::pin(EmptyRecordBatchStream::new(input_schema));
	// continue draining the coalescer
	}
	Some(Ok(batch)) => {
	let timer = elapsed_compute.timer();
	let status = self.predicate.as_ref()
	.evaluate(&batch)
	.and_then(\|v\| v.into_array(batch.num_rows()))
	.and_then(\|array\| {
	Ok(match self.projection.as_ref() {
	Some(projection) => {
	let projected_batch = batch.project(projection)?;
	(array, projected_batch)
	},
	None => (array, batch)
	})
	}).and_then(\|(array, batch)\| {
	match as_boolean_array(&array) {
	Ok(filter_array) => {
	self.metrics.selectivity.add_total(batch.num_rows());
	// TODO: support push_batch_with_filter in LimitedBatchCoalescer
	let batch = filter_record_batch(&batch, filter_array)?;
	let state = self.batch_coalescer.push_batch(batch)?;
	Ok(state)
	}
	Err(_) => {
	internal_err!(
	"Cannot create filter_array from non-boolean predicates"
	)
	}
	}
	})?;
	timer.done();

	match status {
	PushBatchStatus::Continue => {
	// Keep pushing more batches
	}
	PushBatchStatus::LimitReached => {
	// limit was reached, so stop early
	self.batch_coalescer.finish()?;
	// Release the input pipeline's resources.
	let input_schema = self.input.schema();
	self.input =
	Box::pin(EmptyRecordBatchStream::new(input_schema));
	// continue draining the coalescer
	}
	}
	}

	// Error case
	other => return Poll::Ready(other),
	}
	}
	}

RepartitionExec withholds all output from an unbounded input until batch_size rows accumulate #24044

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

jayzhan211 commented on Aug 2, 2026

goutamadwant commented on Aug 8, 2026

calvinchengx commented on Aug 8, 2026

alamb commented on Aug 8, 2026

goutamadwant commented on Aug 8, 2026

goutamadwant commented on Aug 8, 2026

jayzhan211 commented on Aug 9, 2026

calvinchengx commented on Aug 9, 2026

goutamadwant commented on Aug 9, 2026

calvinchengx commented on Aug 9, 2026

ahirner commented on Aug 9, 2026

alamb commented on Aug 11, 2026

alamb commented on Aug 11, 2026

alamb commented on Aug 11, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`RepartitionExec` withholds all output from an unbounded input until `batch_size` rows accumulate #24044