You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Report native operator spill metrics in Spark task metrics for non-shuffle stages #5447
#5370 and #5445 bridge native spill metrics into Spark task metrics (diskBytesSpilled / memoryBytesSpilled), but only for native shuffle write tasks: CometMetricNode.reportSpillMetrics is registered solely on the CometNativeShuffleInputRDD path.
A spill-capable native operator running in a non-shuffle stage (e.g. a sort in a result stage, executed via CometExecRDD in operators.scala) still reports spilled_bytes only as a SQL operator metric. The Spark Stages/task view shows zero spill for the task, the same SQL-view vs task-view discrepancy that #5382 fixed for unified shuffle plans.
Proposed scope
Register a task-level spill reporting callback for native stages executed via CometExecRDD, aggregating over the stage's CometMetricNode tree as sumMetricValues does for shuffle tasks (shared accumulators counted once, disk and memory kept separate, no memory value inferred from disk bytes).
Activity
peterxcli commentedon Aug 24, 2026
take
fix: report native operator spill metrics in Spark task metrics for n…