Repository navigation
Union columns can never be NULL #11162
Description
Activity
@alamb any idea on where I would start looking to try and fix this?
I would suggest writing a standlone test case / reproducer as the first step
Then I suspect we can either help you find the code needed to be fixed (or maybe even someone would be interested in fixing it themselves)
- changed the title
[-]Union columns can never be `NULL` (I think?)[/-][+]Union columns can never be `NULL`[/+]on Jul 7, 2024 See #11314 as a demonstration of the problem for both dense and sparse unions.
After a bit of investigation, the issues lies in the first instance with
datafusion/datafusion/physical-expr/src/expressions/is_null.rs
Lines 74 to 84 in 08c5345
Then with this code in
arrow-rs:/// Returns a non-null [BooleanArray] with whether each value of the array is null. /// # Error /// This function never errors. /// # Example /// ... pub fn is_null(input: &dyn Array) -> Result<BooleanArray, ArrowError> { let values = match input.logical_nulls() { None => BooleanBuffer::new_unset(input.len()), Some(nulls) => !nulls.inner(), }; Ok(BooleanArray::new(values, None)) }
And then with this code
/// Union types always return non null as there is no validity buffer. /// To check validity correctly you must check the underlying vector. fn is_null(&self, _index: usize) -> bool { false }
Ultimately with the spec
Unlike other data types, unions do not have their own validity bitmap. Instead,
the nullness of each slot is determined exclusively by the child arrays which
are composed to create the union.
Basically arrow is saying "we're not going to tell you if a union is null, you need to look in the child arrays", but datafusion isn't listening and is just asking the union if it's null in the naive way.
Two options to move forward as far as I can tell:
- Decide unions in DF can never be null — I'll need to abandon unions in
datafusion-functions-jsonand just return strings everywhere - Have custom logic for unions that looks up the child array to determine if the value is null
If (as I hope) we go for the second option, there's also the issue (as demonstrated by #11314) that the representation of "null" union items doesn't match other types, it shows
{A=}instead of an empty string.- Decide unions in DF can never be null — I'll need to abandon unions in
Have custom logic for unions that looks up the child array to determine if the value is null
+1 for second option. I think we should check the children's nullability.
Reacted by Samuel ColvinI suppose there's a third option of updating arrow-rs to correctly calculate if a
UnionArrayis null, but I presume that works take much longerI've proposed a fix in #11321.
Reacted by Andrew LambI suppose there's a third option of updating arrow-rs to correctly calculate if a
UnionArrayis null, but I presume that works take much longerIt would likely take longer
Note there is a method that takes into account child nullability that perhaps we could use instead of
is_null:
https://docs.rs/arrow/latest/arrow/array/trait.Array.html#method.logical_nullsUpdate: it turns out
logical_nullsis incorrect for UnionArray: apache/arrow-rs#6017
Describe the bug
Maybe I'm doing something wrong with my union in datafusion-functions-json, but
is nullexpressions never evaluate to true with theJsonUnioncolumn.They behave as expected when the function in question returns a scalar
JsonUnion.To Reproduce
See datafusion-contrib/datafusion-functions-json#24
Expected behavior
When the value in every column of the union is null:
<union column> is nullshould evaluate to null{<first column_name>=}(which I what I think it is now)Additional context
No response