Skip to content

Cancel background batch producers before collecting final plan metrics #5504

Closed

Description

@sunchao

Describe the bug

The async I/O execution path discards its producer JoinHandle. When Spark closes a plan early, releasePlan snapshots metrics before the receiver and context are dropped. The producer can still own the stream, so stream-drop metric guards can update counters only after the final snapshot. Closing the consumer also does not explicitly request producer cancellation.

Steps to reproduce

Start an async batch producer whose stream owns a drop-time metric guard and remains pending, then close its consumer before it finishes. A final snapshot taken before the producer releases the stream misses the guard's update. A producer inside synchronous decoding is a distinct case: it cannot observe cancellation until it yields.

Expected behavior

Retain producer ownership, close the receiver, request cancellation, and wait only within a bounded budget before the final metric snapshot. Cooperative completion should expose drop-time updates; timeout must explicitly remain a potentially incomplete snapshot. There must be no claim that a timeout makes external JVM task state safe to tear down.

Additional context

Extracted from #5453 to review cancellation behavior and close latency independently of new I/O counters. Existing producer ownership is async move over the stream and sender, not a captured raw ExecutionContext pointer. Cancellation remains cooperative; this issue does not promise preemption of synchronous decoding or complete accounting after a timeout.

Activity

  1. added
    bugSomething isn't working
    priority:lowMinor issues, test failures, tooling, cosmetic
    area:ffiArrow FFI / JNI boundary
    and removed on Aug 31, 2026
  2. andygrove commented on Sep 26, 2026

    @andygrove
    Member

    This affects memory as well as metrics, which makes it more urgent than priority:low. The producer that releasePlan leaves running still owns the stream's reservations. When a task stops consuming early or fails, Spark's end-of-task cleanup finds them (Managed memory leak detected) and frees them for other tasks while the native buffers are still allocated. The producer releases them a few milliseconds later, and that is the Internal error: release called on ... but task only has 0 bytes warning in #2453. I reproduced it on main, and the details are in #2453 (comment).

    The fix this issue proposes, keeping the handle and having releasePlan cancel and wait within a bound, would fix both. I'm raising the priority and adding area:memory.

  3. added
    area:memoryMemory pools, reservations, OOM handling
    priority:mediumFunctional bugs, performance regressions, broken features
    and removed
    priority:lowMinor issues, test failures, tooling, cosmetic
    on Sep 26, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:ffiArrow FFI / JNI boundaryarea:memoryMemory pools, reservations, OOM handlingbugSomething isn't workingpriority:mediumFunctional bugs, performance regressions, broken features

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions