You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Panic when inserting array literal with NULL element into a table where the matching column has a non-nullable element #12598
I have a custom data sink whose schema contains an array whose element type is non-nullable. If I attempt to insert an array literal with a null element into this data sink via SQL, the process exits with a panic. This behavior can be reproduced using MemTable.
To Reproduce
The following example code can be added to datafusion-examples/src and run via cargo run --example array-non-nullable-element:
use std::sync::Arc;
use arrow::datatypes::{DataType, Field, Schema};
use arrow::record_batch::RecordBatch;
use datafusion::common::Result;
use datafusion::prelude::*;
#[tokio::main]
async fn main() -> Result<()> {
let schema = Arc::new(Schema::new(vec![
Field::new("id", DataType::Int64, false),
Field::new("arr", DataType::new_list(DataType::Int64, false), true),
]));
let ctx = SessionContext::new();
ctx.register_batch("t", RecordBatch::new_empty(Arc::clone(&schema)))?;
let _ = ctx.table("t").await?;
let _ = ctx
.sql("INSERT INTO t VALUES (1, [1, 2, 3]);")
.await?
.collect()
.await?;
let _ = ctx
.sql("INSERT INTO t VALUES (2, [1, NULL, 3]);")
.await?
.collect()
.await?;
Ok(())
}
Running with RUST_BACKTRACE=1 indicates that there is an unwrap() occurring in ListArray::new.
3: core::result::Result<T,E>::unwrap
at /rustc/eeb90cda1969383f56a2637cbd3037bdf598841c/library/core/src/result.rs:1102:23
4: arrow_array::array::list_array::GenericListArray<OffsetSize>::new
at $HOME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-array-53.0.0/src/array/list_array.rs:228:9
5: arrow_cast::cast::list::cast_list_values
at $HOME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-cast-53.0.0/src/cast/list.rs:144:17
6: arrow_cast::cast::cast_with_options
at $HOME/.cargo/registry/src/index.crates.io-6f17d22bba15001f/arrow-cast-53.0.0/src/cast/mod.rs:750:32
7: datafusion_expr_common::columnar_value::ColumnarValue::cast_to
at $DATAFUSION/datafusion/expr-common/src/columnar_value.rs:195:17
Expected behavior
I would expect this query to bubble a DataFusionError up to the call site instead of panicking the process.
Activity
jonahgao commentedon Sep 24, 2024
It looks like an arrow-rs problem,
GenericListArray::try_newshould be used instead ofGenericListArray::new🤔rroelke commentedon Sep 25, 2024
Is the right approach to close this issue after opening a new one against
arrow-rs?jonahgao commentedon Sep 26, 2024
I think we can keep it open and wait for an upstream fix.
Jefffrey commentedon Sep 20, 2025
Upstream fix: apache/arrow-rs#8397
Propagate errors instead of panics: Replace usages of `new` with `try…
Jefffrey commentedon Sep 21, 2025
Merged; pending upgrade to arrow 57.0.0 where fix should come through in
Jefffrey commentedon Dec 10, 2025
Upgrade done:
Now errors instead of panics: