[substrait] [sqllogictest] Null cast is not valid #16272

Activity

1. gabotechs

   added

   bugSomething isn't workingSomething isn't working

   on Jun 6, 2025
2. jkosh44 commented on Aug 16, 2025

   

   jkosh44

   on Aug 16, 2025

   Contributor

   More actions

   take
3. github-actions

   assigned

   jkosh44

   on Aug 16, 2025
4. jkosh44 commented on Aug 16, 2025

   

   jkosh44

   on Aug 16, 2025

   Last edited by jkosh44

   Contributor

   More actions

   Here's a more minimal .slt repro

   query I rowsort
   select column1 from VALUES (1) where column1 = NULL
   ----
   
5. jkosh44 commented on Aug 17, 2025

   

   jkosh44

   on Aug 17, 2025

   Contributor

   More actions

   The failure happens when converting the DF logical plan into substrait. The logical plan of the above query looks like

   Projection(Projection {
       expr: [
           Column(Column {
               relation: None,
               name: "column1"
           })
       ],
       input: Filter(Filter {
           predicate: BinaryExpr(BinaryExpr {
               left: Column(Column {
                   relation: None,
                   name: "column1"
               }),
               op: Eq,
               right: Literal(NULL, None)
           }),
           input: Values(Values {
               schema: DFSchema {
                   inner: Schema {
                       fields: [
                           Field {
                               name: "column1",
                               data_type: Int64,
                               nullable: true,
                               dict_id: 0,
                               dict_is_ordered: false,
                               metadata: {}
                           }
                       ],
                       metadata: {}
                   },
                   field_qualifiers: [None],
                   functional_dependencies: FunctionalDependencies { deps: [] }
               },
               values: [
                   [Literal(Int64(1), None)]
               ]
           })
       }),
       schema: DFSchema {
           inner: Schema {
               fields: [
                   Field {
                       name: "column1",
                       data_type: Int64,
                       nullable: true,
                       dict_id: 0,
                       dict_is_ordered: false,
                       metadata: {}
                   }
               ],
               metadata: {}
           },
           field_qualifiers: [None],
           functional_dependencies: FunctionalDependencies { deps: [] }
       }
   })
   
   

   The conversion that fails is the right hand side of the predicate, specifically right: Literal(NULL, None). We correctly identify that the literal is null.

   datafusion/datafusion/substrait/src/logical_plan/producer/expr/literal.rs

   Lines 54 to 67 in 8f15991

   	pub(crate) fn to_substrait_literal(
   	producer: &mut impl SubstraitProducer,
   	value: &ScalarValue,
   	) -> datafusion::common::Result<Literal> {
   	if value.is_null() {
   	return Ok(Literal {
   	nullable: true,
   	type_variation_reference: DEFAULT_TYPE_VARIATION_REF,
   	literal_type: Some(LiteralType::Null(to_substrait_type(
   	&value.data_type(),
   	true,
   	)?)),
   	});
   	}

   However, when we see that the type of the null is NULL, we return an error.

   datafusion/datafusion/substrait/src/logical_plan/producer/types.rs

   Lines 34 to 44 in 8f15991

   	pub(crate) fn to_substrait_type(
   	dt: &DataType,
   	nullable: bool,
   	) -> datafusion::common::Result<substrait::proto::Type> {
   	let nullability = if nullable {
   	r#type::Nullability::Nullable as i32
   	} else {
   	r#type::Nullability::Required as i32
   	};
   	match dt {
   	DataType::Null => internal_err!("Null cast is not valid"),

   I think for this to work properly, we have to know the actual type of the null, since substrait nulls must be typed. For example if we modify the .slt to the following

   query I rowsort
   select column1 from VALUES (1) where column1 = (NULL::INT)
   ----
   

   it works fine.
6. jkosh44 commented on Sep 26, 2025

   

   jkosh44

   on Sep 26, 2025

   Contributor

   More actions

   I can't figure out how to un-assign myself but I don't think I have the time to continue on this. For whoever picks this up here are some ideas about how to resolve this

   • Add support for untyped NULL to substrait.
   • Perform a pass over the Datafusion logical plan to add type information to untyped NULLs. It's possible that this is already implemented elsewhere in Datafusion and then we can just re-use that logic. Otherwise I think it will be a bit non-trivial to implement this in general.
7. gene-bordegaray commented on Oct 14, 2025

   

   gene-bordegaray

   on Oct 14, 2025

   Last edited by gene-bordegaray

   Contributor

   More actions

   take

   ❤️React with ❤️1Reacted by Gabriel
8. github-merge-queue

   added a commit that references this issue on Nov 3, 2025

   fix: null cast not valid in substrait round trip (#18414)

   ...

   This issue will close once commit 720d4f1 is merged into the 'main' branch.

   720d4f1
9. gabotechs

   closed this as completedin #18414on Nov 3, 2025
10. codetyri0n

    added a commit that references this issue on Nov 11, 2025

    fix: null cast not valid in substrait round trip (apache#18414)

    ...

    This issue will close once commit 82340bb is merged into the 'main' branch.

    82340bb