Repository navigation
Comet's explode is slower than Spark when the element is a struct, worst on nested rows #5685
Description
What is the problem the feature request solves?
CometExplodeExec is slower than Spark's GenerateExec whenever the exploded element is a struct, and the gap is largest on the deeply nested, wide row shapes that motivate a generator in the first place.
Measured with CometExplodeBenchmark (added in #5381) on an Apple M3 Max, JDK 17, local[1], Spark 4.1 / Scala 2.13, release native build. These are whole-query times including the Parquet scan and the counting aggregate, so they understate the operator's share (Best ms):
| Case | Spark | Comet | Relative |
|---|---|---|---|
explode array<struct<a bigint, b string>>[10] |
61 | 86 | 0.7X |
explode array<struct>[4] at depth 1 |
69 | 156 | 0.4X |
explode array<struct>[4] at depth 8 |
84 | 167 | 0.5X |
explode array<struct>[4] at depth 8, then its inner array |
96 | 181 | 0.5X |
For contrast, over array<bigint> at fan-out 10 Comet is 1.2X faster, and at fan-out 2 it is 1.5X faster. So this is not a general generator problem — it tracks the element type.
The nesting itself is cheap. Going from depth 1 to depth 8 costs Comet 11ms and Spark 15ms, and nested schema pruning narrows the scan to exactly the events path either way, so both depths read the same leaf payload. What costs is the element being a wide struct: the depth-8 element here is struct<platform string, entries array<struct<type string, ts bigint, page string, source string>>>, and the plain array<struct<bigint, string>> case shows the same direction at a smaller magnitude.
The nested cases are the most reproducible numbers in the whole benchmark — across five runs Comet's arm varied by a few ms while Spark's swung under machine contention.
Describe the potential solution
ExplodeExec is Comet's fork of DataFusion's UnnestExec, and it gathers the unnested column through a take-index buffer sized to the output. For a struct element that gather recurses into every field, which is where the cost concentrates.
#5667 is in flight and replaces the gather with a slice of the child when the take indices would be a contiguous run, which covers plain explode and both columns of posexplode. Its Rust-level benchmark shows explode_element_type/struct down 91%, so it should close a large part of this. This issue is for whatever remains after that lands, and for the cases #5667 deliberately does not reach:
explode_outer/posexplode_outerstill gather, because a NULL or empty row is padded with a NULL that no slice of the child contains.- Carried passthrough columns are still a real gather.
- Whether a struct element benefits from slicing as much as a primitive one needs measuring rather than assuming, since
ListArray::sliceleavesvalueswhole and a struct's field arrays are sliced independently.
Additional context
The benchmark's nested group deliberately counts the whole event struct rather than one of its fields, so nested schema pruning cannot narrow the exploded element and leave the case measuring a two-column gather.
A generator over a map cannot be measured against Comet at all yet — see #2837 — so the shape a real schema would use for this data (a map keyed by platform) is an array<struct<platform, entries>> in the benchmark.
Activity
andygrove commentedon Sep 4, 2026
#5667 is now rebased on merged
main. I measured what it does to the cases above, so this issue can be scoped to what is left rather than guessed at.CometExplodeBenchmark,mainat719cba110against the rebased branch, both built and run back to back on one machine: Apple M3 Ultra, JDK 17,local[1], Spark 4.1 / Scala 2.13, release native build (Best ms):explode array<struct<a bigint, b string>>[10]explode array<struct>[4]at depth 1explode array<struct>[4]at depth 8explode array<struct>[4]at depth 8, then its inner arrayThe Spark arm moves by at most 3ms between the two runs across the whole benchmark, so it holds as a control and the movement is Comet's.
The flat
array<struct>case, the one in the title, crosses over: 0.9X to 1.1X. The three nested cases improve by 15% to 26% and stay behind Spark at 0.7X to 0.8X. So #5667 closes the first row and narrows the rest without closing them, which is what this issue anticipated. It should stay open for the nested gap.The numbers here are on a different machine from the ones in the issue body, which were an M3 Max, so read the before/after columns against each other rather than against the table above.
One thing outside this issue that the same run turned up:
explode array<bigint>[100]was also behind Spark onmain(148ms against 127, 0.9X) and is 112ms / 1.1X after. That was not visible in the original M3 Max run, where the case was not regressed.andygrove commentedon Sep 4, 2026
I decomposed the depth-1 nested case to find out where the remaining time actually goes, and the answer reframes this issue: the generator is a small part of it. The dominant term is the nested Parquet scan, which is #4859.
All numbers below are the same machine and build as my previous comment (Apple M3 Ultra, JDK 17,
local[1], Spark 4.1 / Scala 2.13, release native, 100K rows of the benchmark'snestedview), on this branch with #5667 applied. Best ms.Stripping the case down
count(k), flat column onlycount(events), scan the whole struct, no generator at allexplode, no carried columnsk, region, which is this issue'sdepth 1caseOf Comet's 113ms, 92ms is already spent before any generator exists.
ExplodeExecadds about 19ms and the carried-column gather about 2ms. Spark's generator adds close to nothing, because whole-stage codegen fuses it into the scan loop, but it is a small absolute number under either engine. Further work on the operator is chasing a fifth of the gap.It is not general scan overhead, it is nested pruning
Comet wins on the flat column, so this is specific to nested reads. And this pair is the tell:
count(events), 5 leavescount(events.platform), 1 leafSpark drops to a third of its time when only one leaf is needed. Comet does not move at all. Both executed plans print the identical pruned
ReadSchema: struct<events:array<struct<platform:string>>>, so the pruning reaches the plan and then has no effect on what Comet reads. That is #4859.What #5262 does to it
Same queries built and run on
apache/df55:count(events)whole structcount(events.platform)one leafcount(events.entries)inner arrayexplodethen count one leafPruning works there, and the whole-struct read also improves where there is nothing to prune, so DF55 helps the nested reader beyond leaf selection alone.
One caveat for anyone repeating this:
df55has noexplode.rs, it predatesCometExplodeExecby 114 commits, so a generator query on that branch falls back to Spark'sGenerateExec. I measured the full nested case at 0.5X there and discarded it as a different code path. Only the scan-only rows above are valid across the two branches.Suggested order
ExplodeExeccompetes with 92ms to 59ms on the whole struct, or 93ms to 13ms on a pruned leaf.unnest_list_array.