DeltaLake integration not working (Python) (FFI Table providers not working)

Describe the bug

After upgrading to deltalake (Python) 0.25.1, this basic example fails. Was working fine before.

from deltalake import DeltaTable, write_deltalake
import pyarrow as pa
from datafusion import SessionContext

arrow_tbl = pa.Table.from_pydict({"x": [1, 2], "y": [2.0, 4.0]})

write_deltalake("./test/scratch", arrow_tbl, mode="overwrite")

dt = DeltaTable("./test/scratch")

ctx = SessionContext()

ctx.register_table_provider("tbl", dt)

print(ctx.table("tbl"))

To Reproduce

No response

Expected behavior

No response

Additional context

No response

cross posted here: delta-io/delta-rs#3221

What error are you seeing?

Nothing. It just silently crashes on the register_table_provider step. Can't debug or anything. Never seen that in Python before.

not sure really relevant, but I'm also having some compilation issue when trying to use deltalake with datafusion in Rust.

Here's what I get:

   Compiling datafusion v44.0.0
   Compiling datafusion-proto v44.0.0
   Compiling deltalake-core v0.24.0
error[E0308]: mismatched types
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deltalake-core-0.24.0/src/kernel/models/actions.rs:567:9
    |
566 |     fn try_from(value: &TableFeatures) -> Result<Self, Self::Error> {
    |                                           ------------------------- expected `Result<ReaderFeatures, strum::ParseError>` because of return type
567 |         ReaderFeatures::try_from(value.as_ref())
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `strum::ParseError`, found a different `strum::ParseError`
    |
    = note: `strum::ParseError` and `strum::ParseError` have similar names, but are actually distinct types
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum-0.26.3/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/git/checkouts/strum-222d21164b266718/cc240c3/strum/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `strum` are being used?

error[E0308]: mismatched types
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/deltalake-core-0.24.0/src/kernel/models/actions.rs:575:9
    |
574 |     fn try_from(value: &TableFeatures) -> Result<Self, Self::Error> {
    |                                           ------------------------- expected `Result<WriterFeatures, strum::ParseError>` because of return type
575 |         WriterFeatures::try_from(value.as_ref())
    |         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ expected `strum::ParseError`, found a different `strum::ParseError`
    |
    = note: `strum::ParseError` and `strum::ParseError` have similar names, but are actually distinct types
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/registry/src/index.crates.io-6f17d22bba15001f/strum-0.26.3/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
note: `strum::ParseError` is defined in crate `strum`
   --> /home/hanbo/.cargo/git/checkouts/strum-222d21164b266718/cc240c3/strum/src/lib.rs:42:1
    |
42  | pub enum ParseError {
    | ^^^^^^^^^^^^^^^^^^^
    = note: perhaps two different versions of crate `strum` are being used?

For more information about this error, try `rustc --explain E0308`.
error: could not compile `deltalake-core` (lib) due to 2 previous errors

Since this seems to be related to datafusion-python moving to that repo to begin triage

@timsaucer

FYI @timsaucer

@alamb

@alamb it's not entirely clear if the issue is in Datafusion FFI or the python binding though

Thanks. I suspect the culprit was a change made in the ffi signature which is when we also added the ability to get the major version for compatibility checking. I am surprised it silently fails. I’ll try to reproduce.

Which version of datafusion-python are you using? If you have 44.0.0 can you please try upgrading to 45? If that solves it, I know what the problem is. (updated based on the version of delta you specified)

@timsaucer

@timsaucer , I upgraded deltalake to 0.25.4 and datafusion to 45.2.0, and the example above now works fine. Thank you!

Navigation Menu

Repository navigation

DeltaLake integration not working (Python) (FFI Table providers not working) #1077

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

DeltaLake integration not working (Python) (FFI Table providers not working) #1077

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

riziles commented on Feb 23, 2025

ion-elgreco commented on Feb 23, 2025

riziles commented on Feb 23, 2025

mag1cfrog commented on Feb 24, 2025

alamb commented on Mar 21, 2025

alamb commented on Mar 21, 2025

ion-elgreco commented on Mar 21, 2025

timsaucer commented on Mar 21, 2025

timsaucer commented on Mar 22, 2025

riziles commented on Mar 22, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions