Skip to content

Panic when FFILanceTableProvider is used with datafusion filter and scalar index #3953

Closed
Closed
Panic when FFILanceTableProvider is used with datafusion filter and scalar index#3953
Assignees
westonpaceBubbleCal

Description

@genemerewether
genemerewether
opened on Jun 6, 2025

Repro - modify https://github.com/lancedb/lance/blob/main/python/python/tests/test_table_provider.py as follows.

  1. Add a scalar index dataset.create_scalar_index("col1", "BTREE")
  2. Filter on that scalar index in datafusion result = ctx.table("ffi_lance_table").filter(dfn.col("col1") == 4).collect()

../../../external/lance/python/python/tests/test_table_provider.py Fatal Python error: Aborted

datafusion                            47.0.0                                    79.1 MiB    pypi   datafusion-47.0.0-cp39-abi3-manylinux_2_17_x86_64.manylinux2014_x86_64.whl
lancedb                               0.23.0                                    85.9 MiB    pypi   lancedb-0.23.0-cp39-abi3-manylinux_2_28_x86_64.whl
pylance                               0.29.0                                    103.5 MiB   pypi   pylance-0.29.0-cp39-abi3-manylinux_2_28_x86_64.whl

Activity

  1. genemerewether commented on Jun 6, 2025

    @genemerewether
    genemerewether
    on Jun 6, 2025
    Author

    @westonpace @renato2099 - I went to this example to repro a failure I'm seeing, where I can get the following backtrace:

    (same trigger - filtering on a scalar-indexed column on a datafusion dataframe obtained through FFILanceTableProvider)

    thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-io/src/local.rs:96:9:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.

thread '<unnamed>' panicked at /root/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-ffi-0.5.0/src/lib.rs:585:33:
FFI future panicked
stack backtrace:
   0: std::panicking::begin_panic
   1: <datafusion_ffi::table_provider::ForeignTableProvider as datafusion_catalog::table::TableProvider>::scan::{{closure}}
   2: datafusion::physical_planner::DefaultPhysicalPlanner::map_logical_node_to_physical::{{closure}}
   3: datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}}
   4: <datafusion::physical_planner::DefaultPhysicalPlanner as datafusion::physical_planner::PhysicalPlanner>::create_physical_plan::{{closure}}
   5: <datafusion::execution::session_state::DefaultQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan::{{closure}}
   6: datafusion::dataframe::DataFrame::create_physical_plan::{{closure}}
   7: datafusion::dataframe::DataFrame::collect::{{closure}}
   8: datafusion_python::utils::wait_for_future
   9: datafusion_python::dataframe::PyDataFrame::collect
  10: datafusion_python::dataframe::PyDataFrame::__pymethod_collect__
  11: pyo3::impl_::trampoline::trampoline
  12: datafusion_python::dataframe::<impl pyo3::impl_::pyclass::PyMethods<datafusion_python::dataframe::PyDataFrame> for pyo3::impl_::pyclass::PyClassImplCollector<datafusion_python::dataframe::PyDataFrame>>::py_methods::ITEMS::trampoline
  13: method_vectorcall_NOARGS
             at /usr/local/src/conda/python-3.12.10/Objects/descrobject.c:454:24
  14: _PyObject_VectorcallTstate
             at /usr/local/src/conda/python-3.12.10/Include/internal/pycore_call.h:92:11
  15: PyObject_Vectorcall
             at /usr/local/src/conda/python-3.12.10/Objects/call.c:325:12
  16: _PyEval_EvalFrameDefault
             at /home/conda/feedstock_root/build_artifacts/python-split_1744322472500/work/build-static/Python/bytecodes.c:2715:19
  17: PyEval_EvalCode
             at /usr/local/src/conda/python-3.12.10/Python/ceval.c:580:21
  18: run_eval_code_obj
             at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1757:0
  19: run_mod
             at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1778:0
  20: pyrun_file
             at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:1674:0
  21: _PyRun_SimpleFileObject
             at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:459:0
  22: _PyRun_AnyFileObject
             at /usr/local/src/conda/python-3.12.10/Python/pythonrun.c:78:0
  23: pymain_run_file_obj
             at /usr/local/src/conda/python-3.12.10/Modules/main.c:361:0
  24: pymain_run_file
             at /usr/local/src/conda/python-3.12.10/Modules/main.c:380
  25: pymain_run_python
             at /usr/local/src/conda/python-3.12.10/Modules/main.c:634
  26: Py_RunMain
             at /usr/local/src/conda/python-3.12.10/Modules/main.c:714
  27: Py_BytesMain
             at /usr/local/src/conda/python-3.12.10/Modules/main.c:768:12
  28: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  29: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  30: <unknown>

  2. genemerewether commented on Jun 6, 2025

    @genemerewether
    genemerewether
    on Jun 6, 2025
    Author

    This also happens on a fresh conda env with deps: pip install lancedb pylance datafusion pandas numpy

  3. genemerewether commented on Jun 6, 2025

    @genemerewether
    genemerewether
    on Jun 6, 2025
    Author

    I don't know if this is helpful, but I built lance and datafusion python bindings, and ran
    rust-gdb --args python -m pytest test_table_provider.py

    13 0x00007fff8dad679e in core::panicking::panic_cannot_unwind () at library/core/src/panicking.rs:323
#14 0x00007fff8e4a1093 in datafusion_ffi::execution_plan::execute_fn_wrapper (plan=0x5bf36172ea8, partition=0) at src/execution_plan.rs:107
#15 0x00007fff821c998e in datafusion_ffi::execution_plan::{impl#9}::execute (self=0x5bf36172dd0, partition=0, _context=...) at src/execution_plan.rs:284
#16 0x00007fff83beee61 in datafusion_physical_plan::execution_plan::execute_stream (plan=..., context=...) at src/execution_plan.rs:885
#17 0x00007fff81f5cc23 in datafusion_physical_plan::execution_plan::collect::{async_fn#0} ()
    at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-physical-plan-47.0.0/src/execution_plan.rs:867
#18 0x00007fff81c237ff in datafusion::dataframe::{impl#2}::collect::{async_fn#0} ()
    at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/datafusion-47.0.0/src/dataframe/mod.rs:1351
#19 0x00007fff81c0372b in core::future::future::{impl#1}::poll<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>> (self=...,
    cx=0x7fffffff7440) at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123
#20 0x00007fff8207b17e in tokio::runtime::park::{impl#4}::block_on::{closure#0}<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>> () at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284
#21 0x00007fff8206ebb8 in tokio::task::coop::with_budget<core::task::poll::Poll<core::result::Result<alloc::vec::Vec<arrow_array::record_batch::RecordBatch, alloc::alloc::Global>, datafusion_common::error::DataFusionError>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>>> (budget=..., f=...) at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:167
#22 tokio::task::coop::budget<core::task::poll::Poll<core::result::Result<alloc::vec::Vec<arrow_array::record_batch::RecordBatch, alloc::alloc::Global>, datafusion_common::error::DataFusionError>>, tokio::runtime::park::{impl#4}::block_on::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>>> (f=...) at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:133
#23 tokio::runtime::park::CachedParkThread::block_on<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>> (
    self=0x7fffffff772e, f=...) at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284
#24 0x00007fff81c41b0f in tokio::runtime::context::blocking::BlockingRegionGuard::block_on<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>> (self=0x7fffffff78f0, f=...) at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/blocking.rs:66
#25 0x00007fff820b83c6 in tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure#0}<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>> (blocking=0x7fffffff78f0)
    at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/mod.rs:87
#26 0x00007fff81ec3e80 in tokio::runtime::context::runtime::enter_runtime<tokio::runtime::scheduler::multi_thread::{impl#0}::block_on::{closure_env#0}<core::pin::Pin<alloc::boxed::Box<datafusion::dataframe::{impl#2}::collect::{async_fn_env#0}, alloc::alloc::Global>>>, core::result::Result<alloc::vec::Vec<arrow_array::record_batch::RecordBatch, alloc::alloc::Global>, datafusion_common::error::DataFusionError>> (handle=0x7fff8a1a2198 <datafusion_python::utils::get_tokio_runtime::RUNTIME+48>, allow_block_in_place=true, f=...)
    at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/runtime.rs:65

  4. timsaucer commented on Jun 6, 2025

    @timsaucer
    timsaucer
    on Jun 6, 2025
    Contributor

    Can you try rolling back to datafusion-python 46 and test again?

  5. genemerewether commented on Jun 6, 2025

    @genemerewether
    genemerewether
    on Jun 6, 2025
    Author

    Still occurs - let me try rolling back in a few other environments. Confirmed across multiple environments, and in the original code

  6. timsaucer commented on Jun 6, 2025

    @timsaucer
    timsaucer
    on Jun 6, 2025
    Contributor

    Thanks. I'll try reproducing this morning

  7. timsaucer commented on Jun 6, 2025

    @timsaucer
    timsaucer
    on Jun 6, 2025
    Contributor

    I am guessing this must have something to do with the various tokio runtimes hanging around. I think the scalar index functions are using the compute runtime on the side, and I'm wondering if there is some kind of problem related to this. I'm trying to track down exactly where the problem is happening still.

  8. eddyxu
    assigned
    genemerewether
    ,
    westonpace
    and
    BubbleCal
    and unassigned
    genemerewether
    on Jun 6, 2025

  9. westonpace commented on Jun 6, 2025

    @westonpace
    westonpace
    on Jun 6, 2025
    Member

    Oh, yeah. This is probably related to apache/datafusion#15691

    We have to make sure all our nodes don't rely on an executor being present during the call to the execute function. If the plan uses a scalar index we probably hit some nodes that haven't been tested for this.

  10. westonpace commented on Jun 6, 2025

    @westonpace
    westonpace
    on Jun 6, 2025
    Member

    Alternatively, we could maybe upstream a fix into datafusion-python to attach the executor during the call to execute.

  11. gene-maniro commented on Jun 6, 2025

    @gene-maniro
    gene-maniro
    on Jun 6, 2025

    @westonpace - anything we can do (even a gross hack) in the meantime? I'm imagining some lance python call I can make that will ensure the right machinery is in place, before accessing through FFILanceTableProvider

  12. timsaucer commented on Jun 6, 2025

    @timsaucer
    timsaucer
    on Jun 6, 2025
    Contributor

    Ok, I did a quick test on entering the runtime during execute and that does resolve the problem. I see in the discussion in apache/datafusion#15691 someone pointed out another example of someone who would run spawn tasks in execute. I can put something together, but since DF48 is already under release vote I suspect it will need to go into DF49. I don't want to rush it and I think we need to carefully consider all of the other places in the FFI work that we will want to enter the runtime beyond just this one.

    This is the entirety of the patch I needed to resolve it (in datafusion):

    diff --git a/datafusion/ffi/src/execution_plan.rs b/datafusion/ffi/src/execution_plan.rs
index 00602474d..d6b292450 100644
--- a/datafusion/ffi/src/execution_plan.rs
+++ b/datafusion/ffi/src/execution_plan.rs
@@ -112,6 +112,7 @@ unsafe extern "C" fn execute_fn_wrapper(
     let plan = &(*private_data).plan;
     let ctx = &(*private_data).context;
     let runtime = (*private_data).runtime.clone();
+    let _guard = runtime.as_ref().map(|rt| rt.enter());

     rresult!(plan
         .execute(partition, Arc::clone(ctx))

    I will open an issue in that repo for tracking upstream. In the mean time if you like I can put up a branch that is 47 (since you're upgraded on main) with just this one patch in, but I don't know how lance feels about carrying around non-released versions like that.

  13. westonpace commented on Jun 6, 2025

    @westonpace
    westonpace
    on Jun 6, 2025
    Member

    I don't want to rush it and I think we need to carefully consider all of the other places in the FFI work that we will want to enter the runtime beyond just this one.

    Fortunately, I think execute is the only outstanding one at the moment. I don't expect we will be launching background tasks in the table provider anytime soon. The only other spot I'm aware of is polling the stream returned by execute and that is already covered (the runtime is attached there).

    I will open an issue in that repo for tracking upstream. In the mean time if you like I can put up a branch that is 47 (since you're upgraded on main) with just this one patch in, but I don't know how lance feels about carrying around non-released versions like that.

    Let me see if I can tackle this today with a fix in Lance. I'm personally not opposed to a (short term) fork while waiting for a release but I'd rather put a workaround if it's not too much effort (and either will require a new lance release)

  14. westonpace commented on Jun 6, 2025

    @westonpace
    westonpace
    on Jun 6, 2025
    Member

    #3954 should resolve this as a workaround while we wait for the df-python change. It turns out it was the exec adding the row-addr column that was the culprit. Are you passing in with_row_addr=True? Do you depend on this? If not, you might be able to avoid the issue (without any lance change) by setting with_row_addr=False?

  15. gene-maniro commented on Jun 7, 2025

    @gene-maniro
    gene-maniro
    on Jun 7, 2025

    Interesting, I'll need to look for a new repro (I'm already setting with_row_addr=False in our code). In the repro above, confirmed that with_row_addr=False succeeds

  16. gene-maniro commented on Jun 7, 2025

    @gene-maniro
    gene-maniro
    on Jun 7, 2025

    Ok, got the (new) backtrace for the actual error using this particular table created by our code (had to modify datafusion-python and lance Cargo.toml for all local deps and rebuild.

    thread '<unnamed>' panicked at /home/gene/external/lance/rust/lance-io/src/local.rs:96:9:
there is no reactor running, must be called from the context of a Tokio 1.x runtime
stack backtrace:
   0: rust_begin_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:695:5
   1: core::panicking::panic_fmt
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:75:14
   2: core::panicking::panic_display
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panicking.rs:261:5
   3: tokio::runtime::scheduler::Handle::current::panic_cold_display
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panic.rs:100:13
   4: tokio::runtime::scheduler::Handle::current
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/scheduler/mod.rs:93:27
   5: tokio::runtime::handle::Handle::current
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/handle.rs:141:20
   6: tokio::runtime::blocking::pool::spawn_blocking
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/runtime/blocking/pool.rs:184:14
   7: tokio::task::blocking::spawn_blocking
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/tokio-1.45.1/src/task/blocking.rs:207:9
   8: lance_io::local::LocalObjectReader::open::{{closure}}::{{closure}}
             at /home/gene/external/lance/rust/lance-io/src/local.rs:96:9
   9: lance_io::local::LocalObjectReader::open::{{closure}}
             at /home/gene/external/lance/rust/lance-io/src/local.rs:88:5
  10: lance_io::object_store::ObjectStore::open_with_size::{{closure}}
             at /home/gene/external/lance/rust/lance-io/src/object_store.rs:439:88
  11: lance_table::io::manifest::read_manifest_indexes::{{closure}}::{{closure}}
             at /home/gene/external/lance/rust/lance-table/src/io/manifest.rs:123:18
  12: lance_table::io::manifest::read_manifest_indexes::{{closure}}
             at /home/gene/external/lance/rust/lance-table/src/io/manifest.rs:113:1
  13: lance::index::<impl lance_index::traits::DatasetIndexExt for lance::dataset::Dataset>::load_indices::{{closure}}
             at /home/gene/external/lance/rust/lance/src/index.rs:482:18
  14: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  15: <lance::dataset::Dataset as lance::index::DatasetIndexInternalExt>::scalar_index_info::{{closure}}::{{closure}}
             at /home/gene/external/lance/rust/lance/src/index.rs:1140:43
  16: <lance::dataset::Dataset as lance::index::DatasetIndexInternalExt>::scalar_index_info::{{closure}}
             at /home/gene/external/lance/rust/lance/src/index.rs:1138:5
  17: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  18: lance::dataset::scanner::Scanner::create_plan::{{closure}}::{{closure}}
             at /home/gene/external/lance/rust/lance/src/dataset/scanner.rs:1381:63
  19: lance::dataset::scanner::Scanner::create_plan::{{closure}}
             at /home/gene/external/lance/rust/lance/src/dataset/scanner.rs:1335:5
  20: <lance::datafusion::dataframe::LanceTableProvider as datafusion_catalog::table::TableProvider>::scan::{{closure}}
             at /home/gene/external/lance/rust/lance/src/datafusion/dataframe.rs:108:28
  21: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/future/future.rs:124:9
  22: datafusion_ffi::table_provider::scan_fn_wrapper::{{closure}}
             at /home/gene/external/datafusion/datafusion/ffi/src/table_provider.rs:266:18
  23: async_ffi::LocalBorrowingFfiFuture<T>::new::poll_fn::{{closure}}::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-ffi-0.5.0/src/lib.rs:536:51
  24: async_ffi::FfiContext::with_context
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-ffi-0.5.0/src/lib.rs:249:9
  25: async_ffi::LocalBorrowingFfiFuture<T>::new::poll_fn::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-ffi-0.5.0/src/lib.rs:536:17
  26: core::ops::function::FnOnce::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/ops/function.rs:250:5
  27: <core::panic::unwind_safe::AssertUnwindSafe<F> as core::ops::function::FnOnce<()>>::call_once
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/core/src/panic/unwind_safe.rs:272:9
  28: std::panicking::try::do_call
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:587:40
  29: __rust_try
  30: std::panicking::try
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panicking.rs:550:19
  31: std::panic::catch_unwind
             at /rustc/05f9846f893b09a1be1fc8560e33fc3c815cfecb/library/std/src/panic.rs:358:14
  32: async_ffi::LocalBorrowingFfiFuture<T>::new::poll_fn
             at /home/gene/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/async-ffi-0.5.0/src/lib.rs:534:19
  33: <async_ffi::LocalBorrowingFfiFuture<T> as core::future::future::Future>::poll::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:582:45
  34: <core::task::wake::Context as async_ffi::ContextExt>::with_ffi_context
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:357:9
  35: <async_ffi::LocalBorrowingFfiFuture<T> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:582:9
  36: <async_ffi::BorrowingFfiFuture<T> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:499:9
  37: <datafusion_ffi::table_provider::ForeignTableProvider as datafusion_catalog::table::TableProvider>::scan::{{closure}}
             at /home/gene/external/datafusion/datafusion/ffi/src/table_provider.rs:439:14
  38: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  39: datafusion::physical_planner::DefaultPhysicalPlanner::map_logical_node_to_physical::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:458:22
  40: datafusion::physical_planner::DefaultPhysicalPlanner::task_helper::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:374:14
  41: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528:17
  42: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  43: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/buffer_unordered.rs:75:15
  44: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.31/src/stream.rs:206:9
  45: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/try_stream/try_collect.rs:46:26
  46: datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:337:14
  47: <datafusion::physical_planner::DefaultPhysicalPlanner as datafusion::physical_planner::PhysicalPlanner>::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:187:14
  48: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  49: <datafusion::execution::session_state::DefaultQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/execution/session_state.rs:1916:14
  50: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  51: datafusion::execution::session_state::SessionState::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/execution/session_state.rs:654:14
  52: datafusion::dataframe::DataFrame::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/dataframe/mod.rs:253:61
  53: datafusion::dataframe::DataFrame::collect::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/dataframe/mod.rs:1350:48
  54: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  55: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284:60
  56: tokio::task::coop::with_budget
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:167:5
  57: tokio::task::coop::budget
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:133:5
  58: tokio::runtime::park::CachedParkThread::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284:31
  59: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/blocking.rs:66:9
  60: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  61: tokio::runtime::context::runtime::enter_runtime
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/runtime.rs:65:16
  62: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  63: tokio::runtime::runtime::Runtime::block_on_inner
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/runtime.rs:358:45
  64: tokio::runtime::runtime::Runtime::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/runtime.rs:328:13
  65: datafusion_python::utils::wait_for_future::{{closure}}
             at /home/gene/external/datafusion-python/src/utils.rs:57:25
  66: pyo3::marker::Python::allow_threads
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/marker.rs:520:9
  67: datafusion_python::utils::wait_for_future
             at /home/gene/external/datafusion-python/src/utils.rs:57:5
  68: datafusion_python::dataframe::PyDataFrame::collect
             at /home/gene/external/datafusion-python/src/dataframe.rs:394:23
  69: datafusion_python::dataframe::PyDataFrame::__pymethod_collect__
             at /home/gene/external/datafusion-python/src/dataframe.rs:204:1
  70: pyo3::impl_::trampoline::noargs::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:34:30
  71: pyo3::impl_::trampoline::trampoline::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:190:54
  72: std::panicking::try::do_call
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:554:40
  73: __rust_try
  74: std::panicking::try
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:518:19
  75: std::panic::catch_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panic.rs:345:14
  76: pyo3::impl_::trampoline::trampoline
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:190:9
  77: pyo3::impl_::trampoline::noargs
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:34:14
  78: datafusion_python::dataframe::<impl pyo3::impl_::pyclass::PyMethods<datafusion_python::dataframe::PyDataFrame> for pyo3::impl_::pyclass::PyClassImplCollector<datafusion_python::dataframe::PyDataFrame>>::py_methods::ITEMS::trampoline
             at /home/gene/external/datafusion-python/src/dataframe.rs:204:1
  79: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:3159:19
  80: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  81: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  82: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788:0
  83: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  84: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  85: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  86: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  87: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788:0
  88: _PyObject_Call
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:367:19
  89: PyCFunction_Call
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:387:12
  90: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:3263:26
  91: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  92: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  93: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788:0
  94: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  95: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  96: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
thread '<unnamed>' panicked at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:585:33:
FFI future panicked
stack backtrace:
   0: std::panicking::begin_panic
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:734:5
   1: <async_ffi::LocalBorrowingFfiFuture<T> as core::future::future::Future>::poll::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:585:33
   2: core::result::Result<T,E>::unwrap_or_else
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/result.rs:1456:23
   3: <async_ffi::LocalBorrowingFfiFuture<T> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:582:9
   4: <async_ffi::BorrowingFfiFuture<T> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/async-ffi-0.5.0/src/lib.rs:499:9
   5: <datafusion_ffi::table_provider::ForeignTableProvider as datafusion_catalog::table::TableProvider>::scan::{{closure}}
             at /home/gene/external/datafusion/datafusion/ffi/src/table_provider.rs:439:14
   6: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
   7: datafusion::physical_planner::DefaultPhysicalPlanner::map_logical_node_to_physical::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:458:22
   8: datafusion::physical_planner::DefaultPhysicalPlanner::task_helper::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:374:14
   9: <futures_util::stream::futures_unordered::FuturesUnordered<Fut> as futures_core::stream::Stream>::poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/futures_unordered/mod.rs:528:17
  10: futures_util::stream::stream::StreamExt::poll_next_unpin
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/mod.rs:1638:9
  11: <futures_util::stream::stream::buffer_unordered::BufferUnordered<St> as futures_core::stream::Stream>::poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/stream/buffer_unordered.rs:75:15
  12: <S as futures_core::stream::TryStream>::try_poll_next
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-core-0.3.31/src/stream.rs:206:9
  13: <futures_util::stream::try_stream::try_collect::TryCollect<St,C> as core::future::future::Future>::poll
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/futures-util-0.3.31/src/stream/try_stream/try_collect.rs:46:26
  14: datafusion::physical_planner::DefaultPhysicalPlanner::create_initial_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:337:14
  15: <datafusion::physical_planner::DefaultPhysicalPlanner as datafusion::physical_planner::PhysicalPlanner>::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/physical_planner.rs:187:14
  16: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  17: <datafusion::execution::session_state::DefaultQueryPlanner as datafusion::execution::context::QueryPlanner>::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/execution/session_state.rs:1916:14
  18: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  19: datafusion::execution::session_state::SessionState::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/execution/session_state.rs:654:14
  20: datafusion::dataframe::DataFrame::create_physical_plan::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/dataframe/mod.rs:253:61
  21: datafusion::dataframe::DataFrame::collect::{{closure}}
             at /home/gene/external/datafusion/datafusion/core/src/dataframe/mod.rs:1350:48
  22: <core::pin::Pin<P> as core::future::future::Future>::poll
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/core/src/future/future.rs:123:9
  23: tokio::runtime::park::CachedParkThread::block_on::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284:60
  24: tokio::task::coop::with_budget
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:167:5
  25: tokio::task::coop::budget
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/task/coop/mod.rs:133:5
  26: tokio::runtime::park::CachedParkThread::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/park.rs:284:31
  27: tokio::runtime::context::blocking::BlockingRegionGuard::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/blocking.rs:66:9
  28: tokio::runtime::scheduler::multi_thread::MultiThread::block_on::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/mod.rs:87:13
  29: tokio::runtime::context::runtime::enter_runtime
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/context/runtime.rs:65:16
  30: tokio::runtime::scheduler::multi_thread::MultiThread::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/scheduler/multi_thread/mod.rs:86:9
  31: tokio::runtime::runtime::Runtime::block_on_inner
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/runtime.rs:358:45
  32: tokio::runtime::runtime::Runtime::block_on
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/tokio-1.45.0/src/runtime/runtime.rs:328:13
  33: datafusion_python::utils::wait_for_future::{{closure}}
             at /home/gene/external/datafusion-python/src/utils.rs:57:25
  34: pyo3::marker::Python::allow_threads
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/marker.rs:520:9
  35: datafusion_python::utils::wait_for_future
             at /home/gene/external/datafusion-python/src/utils.rs:57:5
  36: datafusion_python::dataframe::PyDataFrame::collect
             at /home/gene/external/datafusion-python/src/dataframe.rs:394:23
  37: datafusion_python::dataframe::PyDataFrame::__pymethod_collect__
             at /home/gene/external/datafusion-python/src/dataframe.rs:204:1
  38: pyo3::impl_::trampoline::noargs::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:34:30
  39: pyo3::impl_::trampoline::trampoline::{{closure}}
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:190:54
  40: std::panicking::try::do_call
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:554:40
  41: __rust_try
  42: std::panicking::try
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panicking.rs:518:19
  43: std::panic::catch_unwind
             at /rustc/f6e511eec7342f59a25f7c0534f1dbea00d01b14/library/std/src/panic.rs:345:14
  44: pyo3::impl_::trampoline::trampoline
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:190:9
  45: pyo3::impl_::trampoline::noargs
             at /home/gene/.cargo/registry/src/index.crates.io-6f17d22bba15001f/pyo3-0.24.2/src/impl_/trampoline.rs:34:14
  46: datafusion_python::dataframe::<impl pyo3::impl_::pyclass::PyMethods<datafusion_python::dataframe::PyDataFrame> for pyo3::impl_::pyclass::PyClassImplCollector<datafusion_python::dataframe::PyDataFrame>>::py_methods::ITEMS::trampoline
             at /home/gene/external/datafusion-python/src/dataframe.rs:204:1
  47: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:3159:19
  48: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  49: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  50: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788
  51: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  52: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  53: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  54: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  55: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788
  56: _PyObject_Call
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:367:19
  57: PyCFunction_Call
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:387:12
  58: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:3263:26
  59: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  60: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  61: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788
  62: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  63: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  64: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  65: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  66: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788
  67: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  68: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  69: _PyObject_FastCallDictTstate
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:144:15
  70: _PyObject_Call_Prepend
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:508:24
  71: slot_tp_call
             at /usr/local/src/conda/python-3.12.8/Objects/typeobject.c:8788
  72: _PyObject_MakeTpCall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:240:18
  73: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  74: PyEval_EvalCode
             at /usr/local/src/conda/python-3.12.8/Python/ceval.c:578:21
  75: builtin_exec_impl
             at /usr/local/src/conda/python-3.12.8/Python/bltinmodule.c:1096
  76: builtin_exec
             at /usr/local/src/conda/python-3.12.8/Python/clinic/bltinmodule.c.h:586
  77: cfunction_vectorcall_FASTCALL_KEYWORDS
             at /usr/local/src/conda/python-3.12.8/Objects/methodobject.c:438:24
  78: _PyObject_VectorcallTstate
             at /usr/local/src/conda/python-3.12.8/Include/internal/pycore_call.h:92:11
  79: PyObject_Vectorcall
             at /usr/local/src/conda/python-3.12.8/Objects/call.c:325:12
  80: _PyEval_EvalFrameDefault
             at /croot/python-split_1733933860609/work/build-static/Python/bytecodes.c:2715:19
  81: pymain_run_module
             at /usr/local/src/conda/python-3.12.8/Modules/main.c:300
  82: pymain_run_python
             at /usr/local/src/conda/python-3.12.8/Modules/main.c:627
  83: Py_RunMain
             at /usr/local/src/conda/python-3.12.8/Modules/main.c:713
  84: Py_BytesMain
             at /usr/local/src/conda/python-3.12.8/Modules/main.c:767:12
  85: __libc_start_call_main
             at ./csu/../sysdeps/nptl/libc_start_call_main.h:58:16
  86: __libc_start_main_impl
             at ./csu/../csu/libc-start.c:360:3
  87: <unknown>
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
=================================================================== short test summary info ===================================================================
FAILED python/tests/test_table_provider.py::test_table_loading - pyo3_runtime.PanicException: FFI future panicked

  17. gene-maniro commented on Jun 7, 2025

    @gene-maniro
    gene-maniro
    on Jun 7, 2025

    I haven't been able to create a Lance dataset outside our workflow that triggers this, but here's the only code required if you have our bad table:

        ffi_lance_table2 = FFILanceTableProvider(
        lance.dataset(PATH_TO_LANCE_TABLE)
    )

    ctx.register_table_provider("ffi_lance_table2", ffi_lance_table2)
    (
        ctx.table("ffi_lance_table2")
        .filter(dfn.col("MY_BTREE_INDEXED_STR_COL") == MY_STR_VALUE)
        .filter(dfn.col("MY_BTREE_INDEXED_INT_COL") == 0)
        .collect()
    )

  18. gene-maniro commented on Jun 7, 2025

    @gene-maniro
    gene-maniro
    on Jun 7, 2025

    The last thing I get out of rust-gdb before I can't follow what's happening is in /home/gene/external/datafusion/datafusion/ffi/src/table_provider.rs

    (gdb) n
444	        Ok(Arc::new(plan))
(gdb) p plan
$8 = datafusion_ffi::execution_plan::ForeignExecutionPlan {name: alloc::string::String {vec: alloc::vec::Vec<u8, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<u8, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x4de16033140}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (14), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<u8>}, len: 14}}, plan: datafusion_ffi::execution_plan::FFI_ExecutionPlan {properties: 0x7fff8e4cd7b0 <datafusion_ffi::execution_plan::properties_fn_wrapper>, children: 0x7fff8e4cd860 <datafusion_ffi::execution_plan::children_fn_wrapper>, name: 0x7fff8e4cde10 <datafusion_ffi::execution_plan::name_fn_wrapper>, execute: 0x7fff8e4cda10 <datafusion_ffi::execution_plan::execute_fn_wrapper>, clone: 0x7fff8e4cdf30 <datafusion_ffi::execution_plan::clone_fn_wrapper>, release: 0x7fff8e4cded0 <datafusion_ffi::execution_plan::release_fn_wrapper>, private_data: 0x9c5a40}, properties: datafusion_physical_plan::execution_plan::PlanProperties {eq_properties: datafusion_physical_expr::equivalence::properties::EquivalenceProperties {eq_group: datafusion_physical_expr::equivalence::class::EquivalenceGroup {classes: alloc::vec::Vec<datafusion_physical_expr::equivalence::class::EquivalenceClass, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<datafusion_physical_expr::equivalence::class::EquivalenceClass, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x8}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (0), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<datafusion_physical_expr::equivalence::class::EquivalenceClass>}, len: 0}}, oeq_class: datafusion_physical_expr::equivalence::ordering::OrderingEquivalenceClass {orderings: alloc::vec::Vec<datafusion_physical_expr_common::sort_expr::LexOrdering, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<datafusion_physical_expr_common::sort_expr::LexOrdering, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x4de16045ac0}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (1), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<datafusion_physical_expr_common::sort_expr::LexOrdering>}, len: 0}}, constants: alloc::vec::Vec<datafusion_physical_expr::equivalence::class::ConstExpr, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<datafusion_physical_expr::equivalence::class::ConstExpr, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x10}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (0), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<datafusion_physical_expr::equivalence::class::ConstExpr>}, len: 0}, constraints: datafusion_common::functional_dependencies::Constraints {inner: alloc::vec::Vec<datafusion_common::functional_dependencies::Constraint, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<datafusion_common::functional_dependencies::Constraint, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x8}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (0), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<datafusion_common::functional_dependencies::Constraint>}, len: 0}}, schema: alloc::sync::Arc<arrow_schema::schema::Schema, alloc::alloc::Global> {ptr: core::ptr::non_null::NonNull<alloc::sync::ArcInner<arrow_schema::schema::Schema>> {pointer: 0x4de16086a80}, phantom: core::marker::PhantomData<alloc::sync::ArcInner<arrow_schema::schema::Schema>>, alloc: alloc::alloc::Global}}, partitioning: datafusion_physical_expr::partitioning::Partitioning::RoundRobinBatch(1), emission_type: datafusion_physical_plan::execution_plan::EmissionType::Incremental, boundedness: datafusion_physical_plan::execution_plan::Boundedness::Bounded, output_ordering: core::option::Option<datafusion_physical_expr_common::sort_expr::LexOrdering>::None}, children: alloc::vec::Vec<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, alloc::alloc::Global> {buf: alloc::raw_vec::RawVec<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>, alloc::alloc::Global> {inner: alloc::raw_vec::RawVecInner<alloc::alloc::Global> {ptr: core::ptr::unique::Unique<u8> {pointer: core::ptr::non_null::NonNull<u8> {pointer: 0x4de16014fc0}, _marker: core::marker::PhantomData<u8>}, cap: alloc::raw_vec::Cap (4), alloc: alloc::alloc::Global}, _marker: core::marker::PhantomData<alloc::sync::Arc<dyn datafusion_physical_plan::execution_plan::ExecutionPlan, alloc::alloc::Global>>}, len: 1}}

  19. gene-maniro commented on Jun 7, 2025

    @gene-maniro
    gene-maniro
    on Jun 7, 2025

    Got it - this is the important error when running on our table:

    thread '<unnamed>' panicked at /home/runner/work/lance/lance/rust/lance-io/src/local.rs:96:9:
there is no reactor running, must be called from the context of a Tokio 1.x runtime

    If I do certain lance operations (e.g. prewarm_index) on the dataset before passing to FFI, it works. If not, I get this panic on failing to open from the local filesystem:

    e.g. good:

        dataset2 = lance.dataset(PATH_TO_TABLE)
    dataset2.optimize.compact_files()
    ffi_lance_table2 = FFILanceTableProvider(dataset2)

    bad:

        dataset2 = lance.dataset(PATH_TO_TABLE)
    ffi_lance_table2 = FFILanceTableProvider(dataset2)

    So it's something about opening a scalar-indexed table and passing to FFI immediately.

  20. westonpace commented on Jun 7, 2025

    @westonpace
    westonpace
    on Jun 7, 2025
    Member

    I haven't been able to create a Lance dataset outside our workflow that triggers this, but here's the only code required if you have our bad table:

    If I do certain lance operations (e.g. prewarm_index) on the dataset before passing to FFI, it works. If not, I get this panic on failing to open from the local filesystem:

    Yeah, this one is trickier to work around I think. I'm pretty sure I could reproduce it. The problem is happening during the call to scan (not execute like the previous case). In the unit test (and when the index is pre-warmed) the dataset indices metadata (e.g. the list of what indices are available on what columns) has already been loaded into the cache and so we don't need to read the index info from disk.

    If you always call list_indices on the dataset before you create the FFILanceTableProvider then you should be able to avoid the issue? We will need to fix this one in datafusion-python I think. I'm not sure of an easy way to work around this within lance.

  21. westonpace commented on Jun 10, 2025

    @westonpace
    westonpace
    on Jun 10, 2025
    Member

    I was able to reproduce from this info. It would only happen if the schema was large enough (and/or had enough metadata) to occupy more than one block. Otherwise we end up reading the index info when we read the footer so the later index read doesn't actually trigger an IOP.

    I've updated the test to reproduce the issue and inserted a workaround to always load the indices when we create the table provider (this is a pretty cheap call). These are both now part of #3954

    I've filed apache/datafusion-python#1146 to track the issue on the datafusion-python side.

  22. github-actions
    added
    Stale
    on Nov 5, 2025
  23. Xuanwo
    removed
    Stale
    on Nov 10, 2025

  24. Xuanwo commented on Jul 28, 2026

    @Xuanwo
    Xuanwo
    on Jul 28, 2026
    Member

    Closing as completed by #3954. The Lance-side FFI table-provider plus scalar-index filter panic is fixed and covered. Any remaining upstream DataFusion work can continue separately.

  25. Xuanwo
    closed this as completedon Jul 28, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions