Repository navigation
fix(arrow-cast): support unsafe casting with masked null values - #10726
Open
dhruvxvaishnav wants to merge 5 commits into
Open
dhruvxvaishnav wants to merge 5 commits into
dhruvxvaishnav wants to merge 5 commits into
Conversation
Contributor
|
run benchmark cast_kernels |
This comment was marked as duplicate.
This comment was marked as duplicate.
Jefffrey
reviewed
Aug 19, 2026
| return Arc::clone(array.values()); | ||
| } | ||
|
|
||
| let mut builder = BooleanBufferBuilder::new(array.values().len()); |
Contributor
Author
There was a problem hiding this comment.
Updated to use NullBuffer::expand in mask_fixed_size_list_values. Thanks!
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing fix/cast-unsafe-masked-nulls (df27155) to bb1e6cd (merge-base) diff Run configurationrun benchmark cast_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Do not fail safe: false casts on unconvertible values that are masked as nulls in binary, string, list, struct, and map arrays. Fixes apache#10711
dhruvxvaishnav
force-pushed
the
fix/cast-unsafe-masked-nulls
branch
from
August 20, 2026 05:35
77ba73c to
759bb33
Compare
Contributor
|
run benchmark cast_kernels |
This comment was marked as duplicate.
This comment was marked as duplicate.
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing fix/cast-unsafe-masked-nulls (2537e2a) to db915ec (merge-base) diff Run configurationrun benchmark cast_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
Jefffrey
reviewed
Sep 25, 2026
Jefffrey
left a comment
Jefffrey
left a comment
Contributor
There was a problem hiding this comment.
sorry it took me a while to get back to reviewing this
| .build(); | ||
| match data { | ||
| Ok(d) => make_array(d), | ||
| Err(_) => Arc::clone(array.values()), |
Contributor
There was a problem hiding this comment.
i think we should propagate the error here instead of silently returning original
Comment on lines
+56
to
+71
| let mut builder = BooleanBufferBuilder::new(array.values().len()); | ||
| let offsets = array.value_offsets(); | ||
| for i in 0..array.len() { | ||
| let is_valid = array.is_valid(i); | ||
| let len = offsets[i + 1].as_usize() - offsets[i].as_usize(); | ||
| builder.append_n(len, is_valid); | ||
| } | ||
| if builder.len() < array.values().len() { | ||
| builder.append_n(array.values().len() - builder.len(), false); | ||
| } | ||
| let parent_nulls = NullBuffer::from(builder.finish()); | ||
| let combined_nulls = NullBuffer::union(Some(&parent_nulls), array.values().nulls()); | ||
| if combined_nulls.as_ref() != array.values().nulls() { | ||
| let data = array | ||
| .values() | ||
| .to_data() |
Contributor
There was a problem hiding this comment.
i wonder if we can use the nullif kernel here actually. we can skip it for fixedsizelist since that is a simple nullbuffer replacement
and if we can use nullif here we can probably use it for list view, maps, structs, etc.
| let values = array.as_fixed_size_list().values(); | ||
| cast_with_options(values, to, cast_options) | ||
| let array = array.as_fixed_size_list(); | ||
| let values = mask_fixed_size_list_values(array); |
Contributor
There was a problem hiding this comment.
we should probably mask only on cast failure (and then retry the cast); this avoids affecting the fast (optimistic) path
and consider we only need to do this masking for safe: false casting
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
4 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
safe: falsecasting doesn't account for masked null values #10711Rationale for this change
When casting with
safe: false, errors were raised for invalid values even if those values were masked by nulls in the array's null bitmap or child offsets (e.g. invalid UTF-8 in a null binary slot, invalid castable child items in a null fixed-size list, list, list view, struct, or map). Because masked values are logically null, the cast should succeed and preserve the null mask.What changes are included in this PR?
arrow-cast/src/cast/string.rs: Incast_binary_to_stringandcast_binary_view_to_string_view, if fast-path UTF-8 conversion fails withsafe: falseand the array contains nulls, only validate non-null values for valid UTF-8.arrow-cast/src/cast/list.rs: Propagate parent list nulls to child arrays before casting incast_single_element_fixed_size_list_to_values,cast_list_values,cast_list_view_values,cast_list,cast_list_view, andcast_list_to_list_viewwhen the target field is nullable.arrow-cast/src/cast/mod.rs: Propagate parent nulls forFixedSizeListandStructArraycolumns before callingcast_with_optionswhen the target field is nullable.arrow-cast/src/cast/map.rs: Propagate parent nulls to map key and value child arrays when target fields are nullable.arrow-castcovering all container types and binary-to-string casts with masked null values.BufferBuilderimport inherited frommainthat caused the Arrow, Parquet, and Flight Clippy jobs to fail.Are these changes tested?
Yes, new unit tests added in
arrow-cast/src/cast/mod.rs:test_binary_to_string_unsafe_masked_nullstest_binary_view_to_string_view_unsafe_masked_nullstest_fixed_size_list_unsafe_masked_nullstest_list_unsafe_masked_nullstest_list_view_unsafe_masked_nullstest_struct_unsafe_masked_nullstest_map_unsafe_masked_nullsThe following checks pass:
cargo test -p arrow-cast— 382 unit tests and 11 doctests passedcargo clippy -p arrow-string --all-targets --all-features -- -D warningscargo clippy --workspace --all-targets --all-features -- -D warningscargo +stable fmt --all -- --checkgit diff --checkAre there any user-facing changes?
No breaking changes to public APIs.
safe: falsecasting now correctly succeeds on arrays where unconvertible values are masked by nulls instead of erroneously failing.AI assistance
OpenAI Codex assisted with codebase exploration, implementation support, test preparation, validation, review follow-up, and CI diagnosis. The contributor remains responsible for understanding and maintaining the changes.