Repository navigation
Ballista: Finish implementing shuffle mechanism #707
Description
Is your feature request related to a problem or challenge? Please describe what you are trying to do.
The shuffle mechanism in Ballista is not yet working and there are some design issues that need resolving in order to fix this.
When running TPC-H query 12 I see query plans like this being executed:
ShuffleWriterExec: None
ParquetExec: ...
The output from each stage is not read by the next stage. The next stage is redundantly executing the previous stages again.
Describe the solution you'd like
Get shuffles working.
Describe alternatives you've considered
None
Additional context
None
Activity
andygrove commentedon Jul 11, 2021
It turns out there is a fundamental bug here. I am working on fixing this.
[-]Ballista: Avoid pointless shuffle write of Parquet partitions[/-][+]Ballista: Finish implementing shuffle mechanism[/+]