Repository navigation
perf: Adopt bulk-NULL builders and NullBuffer::union_many in datafusion-spark #22454
Copy link
Copy link
Open
Description
Is your feature request related to a problem or challenge?
Two patterns from @neilconway's recent perf PRs are not yet applied throughout the datafusion-spark crate:
- Bulk-NULL builders (#21789, #21849, #21863, #21847, #21854, #21877, #21991, #22029, #21519, #21366) — when an output null mask equals (or is a
NullBuffer::union[_many]of) the input null masks, precompute it once instead of maintaining it row-by-row withappend_null/BooleanBufferBuilder.append. NullBuffer::union_many(#22070) — replaces chains of 3+ binaryNullBuffer::unioncalls with one clone + in-place&=.
An audit of the Spark crate turned up six sites where these patterns apply.
Describe the solution you'd like
Apply the patterns at the following call sites. Each checkbox is one PR.
-
datafusion/spark/src/function/math/width_bucket.rs:206-276, 313-432—width_bucketkernel has four nullable inputs (value, min, max, n_buckets). The macro generates the same code for Float64, Duration, IntervalYearMonth, IntervalMonthDayNano. PrecomputeNullBuffer::union_many([v.nulls(), min.nulls(), max.nulls(), n.nulls()])outside the loop, append placeholders for masked rows, reattach at the end. -
datafusion/spark/src/function/array/slice.rs:152-186—calculate_start_endruns twoInt64Builders in lockstep, bothappend_nullwhen any of (values, start, length) is null. Precomputeunion_manyof three input nulls, fill placeholders, reattach to both builders. -
datafusion/spark/src/function/string/substring.rs:371-401—SparkSubstring's main loop. Three-input null union (array + start_array + optional length_array). Negative-length branch still needs row-levelappend_empty, but the null mask itself can be precomputed. -
datafusion/spark/src/function/datetime/make_dt_interval.rs:170-197— 1-4 nullable inputs wrapped inOption<&ArrayRef>.union_manyover theSome(...)arrays' nulls. -
datafusion/spark/src/function/datetime/make_interval.rs:190-— same structure asmake_dt_interval. -
datafusion/spark/src/function/string/format_string.rs:126-157— accumulatesVec<String>thenStringArray::from(vec). Switch toGenericStringBuilder+append_value(&str). Matches theappend_withpattern from perf: Addappend_withto string builders, use inreplace#22029. -
datafusion/spark/src/function/conversion/cast.rs:199-209, 226-236—cast_int_to_timestamp/cast_float_to_timestampdo per-rowappend_null/append_valuewhere output nulls equal input nulls. Clone input nulls.
Describe alternatives you've considered
Bundling all sites into a single PR. Splitting per file keeps each PR reviewable, especially the width_bucket macro change which touches four type variants.
Additional context
Already use these patterns, skip:
datafusion/spark/src/function/spark/null_utils.rs:63— usesunion_many.datafusion/spark/src/function/map/str_to_map.rs:212— usesunion_many.datafusion/spark/src/function/hash/utils.rs— fast-path null-count checks.
Considered and rejected:
datafusion/spark/src/function/string/elt.rs:115-139— output null mask is data-dependent (which column's nulls to include depends on the runtimeidxvalue).
Reactions are currently unavailable
Metadata
Metadata
Assignees
Labels
No labels
Activity
neilconway commentedon May 22, 2026
Awesome! In addition to the bulk-NULL API and
union_many, we can also make use ofappend_withandappend_byte_map.Using these APIs will require either making them
pub, or moving them to a shared crate likedatafusion-common. My thought was to do the latter, once the open PRs are landed (#22171, #22182, #22283)There's also #22148