You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Build spark_size LargeList lengths as Int32 without Int64 cast #5272
Follow-up from #5233 spark_size for LargeList reuses Arrow's length kernel, which returns Int64, then cast_with_options(..., Int32, safe: false), then (when nulls are present) to_vec() to patch null slots to -1. That is three allocations on the hot path.
On the array_size bench after #5233, LargeList (10% null) is only ~1.3x faster than main (7.54 µs vs 9.70 µs), while no-null List is ~12x. The gap is the Int64 → cast → copy chain, not the kernel idea itself.
Describe the potential solution
Build an Int32 length array straight from the LargeList i64 offset buffer (offsets[i+1] - offsets[i], with a checked conversion that errors on overflow rather than wrapping or becoming -1), then apply the same null → -1 rewrite as the List path (null_count == 0 fast path; otherwise into_parts + set_indices).
Avoid allocating the intermediate Int64 length output and the Int32 cast result when those are only used to derive i32 sizes
Additional context
Bench coverage already exists: spark_size: LargeList (10% null) in benches/array_size.rs. Consider adding a no-null LargeList shape for parity with the List production path.
Keep safe: false / checked overflow semantics: a length that does not fit in i32 must error, not become null then -1
Activity
0lai0 commentedon Aug 5, 2026
take