Repository navigation
Spike: evaluate if cuDF can be used with datafusion-python #936
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
As other DataFrame libraries start moving to leveraging GPU resources, it would be useful to see if we could leverage the work already done in pandas and polars for interoperating with cuDF to give a similar experience in DataFusion.
Describe the solution you'd like
Evaluate the level of effort and technical limitations to using cuDF to evaluate DataFrames. Also worth evaluating is their c++ interface which we could potentially bring in to DataFusion upstream if we are willing to write the appropriate wrappers.
Describe alternatives you've considered
Leave as is.
Additional context
This task is really just focused on researching what would be required and if there is an opportunity here.
Activity
andygrove commentedon Oct 28, 2024
I have some experience in this area. While at NVIDIA, I created a POC with Rust bindings around cuDF and then provided interoperability with arrow-rs. Unfortunately, that code was internal and not open-source. I used cxx to create the bindings.
This repo (datafusion-python) once contained a prototype of translating DataFusion logical plan to cuDF operations (all in Python). It is still there in the history somewhere.
andygrove commentedon Oct 28, 2024
I see that there is now one RAPIDS library that provides Rust bindings: https://docs.rapids.ai/api/cuvs/nightly/rust_api/ so it may be interesting to see what approach they took to wrap C++ in this case.
edit: cuvs is using bindgen
drauschenbach commentedon Nov 5, 2024
Possibly #602.
deanm0000 commentedon Mar 14, 2025
polars has a gpu engine mode. Looks like NVIDIA/RAPIDS made it https://docs.rapids.ai/api/cudf/stable/cudf_polars/ and the meat and potatoes aren't in the main polars library. When I click on Show Source from that page it doesn't go to actual source code so maybe not actually helpful so sorry if this is misdirection.
paleolimbot commentedon May 28, 2025
Just two things I was involved in that may be useful here:
cudf::from_arrow(): https://github.com/rapidsai/cudf/blob/2789fa83d943649b982493d68bbba852f848d82c/cpp/include/cudf/interop.hpp#L677-L698Probably integrating at a higher level than those makes sense (e.g., converting expressions), since the trick of optimizing data transfer to/from the GPU is sort of insane.
The nanoarrow version also includes a (probably oudated) Apple Metal backend, which is potentially easier to integrate because you can use the same memory addresses space for the buffers (or I could, the last time I tried this).
milenkovicm commentedon Jul 28, 2025
polars blog gives quite a lot of details about CUDA integration https://pola.rs/posts/gpu-engine-release/ . Looking the code, looks like they have integrated it in the python layer not in rust.
milenkovicm commentedon Jul 28, 2025
looks like they have used bindgen https://github.com/rapidsai/cuvs/blob/1155a3a427cd1d1bfaf8fe74a937ed6dfa797ec7/rust/cuvs-sys/build.rs#L75 to generate rust bindings
JigaoLuo commentedon Jul 28, 2025
As an external contributor to cuDF, I ran into this issue. I could confirm that cuDF and Polars interact exclusively at the Python level, as there’s currently no binding between cuDF’s C++ backend and Polars' Rust core.
One point worth highlighting is the role of pylibcudf: https://docs.rapids.ai/api/cudf/stable/pylibcudf/. This Python wrapper aims to expose the entire libcudf API with minimal overhead, enabling high-performance data operations in Python. Its design allows efficient bridging between Python and C++ without incurring the usual latency associated with Python bindings. However, pylibcudf still lacks full coverage of libcudf’s functionality, and its documentation is not complete.