Repository navigation
[Epic] Implement support for StringView in DataFusion #10918
Description
Is your feature request related to a problem or challenge?
StringView / BinaryView were added to the Arrow format that make it more suitable for certain types of operations on strings. Specifically when filtering with string data, creating the output StringArray requires copying the strings to a new, packed binary buffer.
GenericByteViewArrary was designed to solve this limitation and the arrow-rs implementation, tracked by apache/arrow-rs#5374, is now complete enough to start adding support into DataFusion
I think we can improve performance in certain cases by using StringView (this is also described in more details in the Pola.rs blog post)
- Reading strings / binary from Parquet files as
StringViewArray/BinaryViewArrayrather thanStringArray/BinaryArraysaves a copy (and @ariesdevil is quite close to having it integrated into the parquet reader ImplementStringViewArrayandBinaryViewArrayreading/writing in parquet arrow-rs#5530) - Evaluating predicates on string expressions (for example
substr(url, 4) = 'http') as the intermediate result ofsubstrcan be called without copying string values
Describe the solution you'd like
I would like to support StringView / BinaryView support in DataFusion.
While my primary usecase is for reading data from parquet, I think teaching DataFusion to use StringView (at least as intermediate values when evaluating expressions may help significantly
Development branch: string-view
Since this feature requires upstream arrow-rs support apache/arrow-rs#5374 that is not yet released we plan to do development on a string-view feature branch:
https://github.com/apache/datafusion/tree/string-view
Task List
Here are some high level tasks (I can help flesh these out for anyone who is interested in helping)
- Implement
arrow_castsupport forStringViewandBinaryView#10920 - Implement equality
=and inequality<>support forStringView#10919 - Improve filter predicates with
Utf8Viewliterals #10998 - Implement
LIKEfor StringView arrays #11024 - Implement equality = and inequality <> support for
BinaryView#10996 - use StringViewArray when reading String columns from Parquet #10921
- Create a branch for StringView / BinaryView development #10961
- Implement
LIKEfor StringView arrays #11024 - Implement
REGEXP_REPLACEfor StringView #11025 - Support for
LargeStringandLargeBinaryforStringViewandBinaryView#11023 -
likebenchmark for StringView arrow-rs#5936 - Add generic coercion rule from
Utf8View->Utf8/BinaryView->Binaryfor compatibility - File tickets for adding native Utf8View / BinaryView support to other functions which would benefit
- Establish pattern for implementing optimized string functions for StringViewArray
- Enable reading StringView by default from Parquet (
schema_force_string_view) by default #11682 - Implement native
StringViewsupport for CharacterLength #11677
Describe alternatives you've considered
No response
Additional context
Polars implemented it recently in rust so that can serve as a motivation
Blog Post https://pola.rs/posts/polars-string-type/
https://twitter.com/RitchieVink/status/1749466861069115790
Facebook/Velox's take: https://engineering.fb.com/2024/02/20/developer-tools/velox-apache-arrow-15-composable-data-management/
Related PRs:
pola-rs/polars#13748
pola-rs/polars#13839
pola-rs/polars#13489
Activity
alamb commentedon Jun 14, 2024
I think we should aim for a first "milestone" of showing improvements for some clickbench queries
jayzhan211 commentedon Jun 19, 2024
Will we completely change StringArray to StringViewArray in Datafusion?
While I try to utilize StringViewArray in #10976 , I found there is schema mismatched issue UTF8 vs UTF8View. To avoid converting StringViewArray to StringArray, we might need to change the schema to UTF8View overall from logical plan to physical plan. If we need to keep both String and StringView, then we need to think about how to deal with the conversion between these two types.
A more concrete example is
We have the string column
bas StringArray and DataType::Utf8 now. Should we convert it to StringViewArray and DataType::Utf8View?If not, if we somehow want to utilize StringViewArray, how do we minize the cost of conversion between String and StringView?
It seems Polars completely refactor their String to StringView 🤔
alamb commentedon Jun 19, 2024
I think since they are two separate types in Arrow we couldn't fully switch to StringView the way polars could as it controls the whole stack. Users could still feed DataFusion StringViewArray from custom TableProviders and would expect StringView at the output.
However what I think we could do is internally to DataFusion (e.g. within the plan, before the final output) is use StringView in the batches that flow through intermediate nodes in the plan.
Indeed, As you point out, I don't think we can transparently switch to using StringView -- instead we would have to start encoding information in the plans about the new types.
I wonder if we could have a new logical optimzier pass that tried to annotate operations that support it to use StringView in their schema rather than String. Then the ExecutionPlans would know if they were supposed to generate StringView as output or the more traditional StringArray 🤔
Here is an idea of one place to start: #9403 (comment)
alamb commentedon Jun 19, 2024
I think @XiangpengHao is looking into another place to use StringView which is #10921 -- where we have a similar idea to use StringView in some sub portion of the plan. Here is more info about the optimizer pass idea: #10921 (comment)
alamb commentedon Jun 20, 2024
I think this project is going pretty well
We are at the point of starting to implement some basic functions using StringView.
LIKEfor StringView arrays #11024REGEXP_REPLACEfor StringView #11025alamb commentedon Jul 9, 2024
Now that we have upgraded to arrow 52.1.0, I think we could merge the
string-viewbranch to main. I'll try and make a PR if no one beats me to italamb commentedon Jul 15, 2024
An update here on the plan from @XiangpengHao:
52.2.0) that show significant performance improvements for TPCH and ClickBench queries52.2.0is released (early August 2024) we'll be able to add an option that makes DataFusion use StringView when reading from Parquet / filteringI for one am very excited
alamb commentedon Jul 16, 2024
Update: we merged the basic support for StringView in #11402
I have created a branch to collect any changes that rely on the pre-release parquet/arrow
52.2.0version here: https://github.com/apache/datafusion/tree/string-view2alamb commentedon Jul 22, 2024
Update here is that we hope to have enough code merged / implemented to report benchmark results by the end of the week. Once arrow-rs 52.2.0 is released apache/arrow-rs#5998 then we'll merge the contents of https://github.com/apache/datafusion/tree/string-view2 to main
Once we do that we'll evaluate what, if anything, is left before we can close this ticket
alamb commentedon Jul 29, 2024
Update: the next major PR in this sequence is ready to merge: #11667
alamb commentedon Jul 31, 2024
Update here is that we have all the basic support in now but it is behind a feature flag. Let's declare this project a success for now and track the remaining work to enable this feature by default in a separate issue: #11752
Thanks to everyone who has helped so far. We are close ™️
alamb commentedon Aug 16, 2024
In case anyone wants an overview of adding StringView to DataFusion, here is a presentation and slides from @XiangpengHao
Video: https://www.youtube.com/watch?v=RVLshX6fbds
Slides: https://drive.google.com/file/d/1Qqd8V6cfS9rSQ_-JrinasQJwI79qlUEV/view?usp=drive_link
feat(spark): implement `StringView` for `SparkConcat` (#19984)
feat(spark): implement `StringView` for `SparkConcat` (apache#19984)