Repository navigation
Convert Utf8View/BinaryView --> Utf8 / Binary at output #12119
Description
Is your feature request related to a problem or challenge?
Part of #11752
We are trying to change DataFusion to use StringViewArray by default when reading parquet (and, for example, when it makes more sense such as the substr function), StringView enables many interesting optimization opportunities. However, as StringView is still being adopted across the rest of the arrow ecosystem, if DataFusion begins to emit StringViewArray in some places, it may cause issues with other parts of the ecosystem (e.g. flight clients may not be able to interpret data sent by a server using DataFusion)
Describe the solution you'd like
I would like DataFusion to retain maximum compatibility at the interfaces, but be able to use StringViewArray internally when it improves performance
Describe alternatives you've considered
I recommend a config flag that makes it possible to convert Utf8View/BinaryView --> Utf8 / Binary at the query output and I think this conversion should be done by default.
For example we might add this configuration flag:
datafusion.optimizer.expand_views_at_output=true
If this flag is true,
- add code in the Analyzer (maybe in the TypeCOercion code)
- check the output columns of a plan, and if any are
DataType::Utf8VieworDataType::BinaryView, add ProjectionExecthat converts them to Utf8/Binary (by adding a cast toDataType::Utf8orDataType::Binary` respectively
Additional context
We already have to do something similar in flight with dictionary arrays
Activity
findepi commentedon Aug 23, 2024
Sounds reasonable.
One question -- do we need a flag though?
alamb commentedon Aug 24, 2024
My rationale is that converting from Utf8View --> Utf8 is not free. Thus users should have the option of not paying the cost if they want performance over compatibility.
I will admit I don't have a specific usecase in mind
alamb commentedon Aug 24, 2024
BTW with this conversion in place, I think we could contemplate more interesting changes, like
substralways outputingUtf8Vieweven when the input wasUtf8findepi commentedon Aug 24, 2024
it is not
but transmitting non-compacted string views isn't free either.
also, we're transitioning from a state where DF didn't use string views (eager compaction) to a state where DF uses string views (deferred compaction). Thus, if we convert to non-view types on output, we do not risk regressing anything. And we can revisit later whether returning view types could be an improvement (and introduce a flag, if needed & worth it).
[-]Add config flag to convert `Utf8View`/`BinaryView` --> `Utf8` / `Binary` at output[/-][+]Convert `Utf8View`/`BinaryView` --> `Utf8` / `Binary` at output[/+]alamb commentedon Aug 26, 2024
Right -- I agree so this is why in my mind having the default be "Utf8" is important
I would not be opposed to a PR that just hard codes the existing behavior (convert to Utf8 on output). I changed the title of this PR to reflect this.
I still think a config option offers the most flexibility but I do agree we could add it later if needed
wiedld commentedon Aug 27, 2024
@findepi -- are you already working on this ticket? If not, I would like to pick it up.
findepi commentedon Aug 27, 2024
@wiedld awesome, go for it
wiedld commentedon Aug 28, 2024
take