Skip to content

Ray shuffle not working  #10

Closed
Closed

Description

@austin362667

I got an error on Mac: Memory pointer from external source (e.g, FFI) is not aligned with the specified scalar type. Before importing buffer through FFI, please make sure the allocation is aligned. while using ray shuffle on Datafusion Ray against TPC-H.

After that the Ray worker died.

@andygrove saw this error when trying to run the example on Linux, but it worked on Mac.

@franklsf95 Any suggestion would be helpful! I appreciate it!

Activity

  1. andygrove commented on Oct 4, 2024

    @andygrove
    Member

    Possible workaround in apache/arrow#43552

  2. andygrove commented on Oct 4, 2024

    @andygrove
    Member

    arrow-rs already has a fix for this in apache/arrow-rs#6472 and is included in 53.1.0, so this should be fixed when we upgrade to a version of DataFusion that uses this Arrow version

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions