You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Cancel background batch producers before collecting final plan metrics #5504
The async I/O execution path discards its producer JoinHandle. When Spark closes a plan early, releasePlan snapshots metrics before the receiver and context are dropped. The producer can still own the stream, so stream-drop metric guards can update counters only after the final snapshot. Closing the consumer also does not explicitly request producer cancellation.
Steps to reproduce
Start an async batch producer whose stream owns a drop-time metric guard and remains pending, then close its consumer before it finishes. A final snapshot taken before the producer releases the stream misses the guard's update. A producer inside synchronous decoding is a distinct case: it cannot observe cancellation until it yields.
Expected behavior
Retain producer ownership, close the receiver, request cancellation, and wait only within a bounded budget before the final metric snapshot. Cooperative completion should expose drop-time updates; timeout must explicitly remain a potentially incomplete snapshot. There must be no claim that a timeout makes external JVM task state safe to tear down.
Additional context
Extracted from #5453 to review cancellation behavior and close latency independently of new I/O counters. Existing producer ownership is async move over the stream and sender, not a captured raw ExecutionContext pointer. Cancellation remains cooperative; this issue does not promise preemption of synchronous decoding or complete accounting after a timeout.
This affects memory as well as metrics, which makes it more urgent than priority:low. The producer that releasePlan leaves running still owns the stream's reservations. When a task stops consuming early or fails, Spark's end-of-task cleanup finds them (Managed memory leak detected) and frees them for other tasks while the native buffers are still allocated. The producer releases them a few milliseconds later, and that is the Internal error: release called on ... but task only has 0 bytes warning in #2453. I reproduced it on main, and the details are in #2453 (comment).
The fix this issue proposes, keeping the handle and having releasePlan cancel and wait within a bound, would fix both. I'm raising the priority and adding area:memory.
Activity
andygrove commentedon Sep 26, 2026
This affects memory as well as metrics, which makes it more urgent than
priority:low. The producer thatreleasePlanleaves running still owns the stream's reservations. When a task stops consuming early or fails, Spark's end-of-task cleanup finds them (Managed memory leak detected) and frees them for other tasks while the native buffers are still allocated. The producer releases them a few milliseconds later, and that is theInternal error: release called on ... but task only has 0 byteswarning in #2453. I reproduced it on main, and the details are in #2453 (comment).The fix this issue proposes, keeping the handle and having
releasePlancancel and wait within a bound, would fix both. I'm raising the priority and addingarea:memory.