Repository navigation
Read string as Utf8View in sqllogictest #13408
Description
Is your feature request related to a problem or challenge?
This is the simplest way to create a string column
statement ok
create table t(a varchar) as values ('1'), ('2');
query T
select arrow_typeof(a) from t;
----
Utf8
Utf8
statement ok
drop table t
I'm finding a simple way to create Utf8View column. We can achieve this with casting (See datafusion/sqllogictest/test_files/string/string_view.slt) but I think we can have a more simpler way to do this.
Describe the solution you'd like
I have two idea but not sure which one is better
Approach 1
Set the configuration with use_string_view = true, and we will read the column (varchar) as string view type instead of string type.
Something like
set datafusion.xxx.use_string_view = true
statement ok
create table t(a varchar) as values ('1'), ('2');
query T
select arrow_typeof(a) from t;
----
Utf8View
Utf8View
statement ok
drop table t
This avoid adding new syntax.
Approach 2
introduce string_view syntax like varchar_view so we know we want to read it as string view type
Something like
statement ok
create table t(a varchar_view) as values ('1'), ('2');
query T
select arrow_typeof(a) from t;
----
Utf8View
Utf8View
statement ok
drop table t
This adds complexity of understanding of string view type for people from Postgres.
Describe alternatives you've considered
No response
Additional context
No response
Activity
[-]Configure string view as default type in sqllogictest[/-][+]Read string as Utf8View in sqllogictest[/+]jayzhan211 commentedon Nov 14, 2024
Now I would prefer the first approach since string view type is more to a "physical type" concept, add this to parser doesn't fit very well.
Omega359 commentedon Nov 15, 2024
The first approach 100%
Weijun-H commentedon Nov 16, 2024
vote for 1st approach
findepi commentedon Nov 16, 2024
if the table
tis created internally using Utf8View, then i agree that SLT should return that on query output.if the table
tis created internally using Utf8, SLT should continue to read it as Utf8, unless DF engine generally replaces Utf8View with Utf8 (which i think it does not)So, do we need a new config after all?
alamb commentedon Dec 17, 2024
The first approach 100%
Likewise, I think the first approach is good
alamb commentedon Jan 27, 2025
I am not sure if you have seen the great
stringpatter in sqllogictest from @goldmedal yet: https://github.com/apache/datafusion/tree/main/datafusion/sqllogictest/test_files/stringBasically it lets you write one set of tests and then verify those tests work with all the different types of string array (Utf8, Utf8View, LargeUtf8View, Dictionary(...), etc)
If this ticket is just about making it easier to write slt tests, I think the
stringtests might be enoughHowever, at least part of this ticket seems to be suggesting changing what Arrow DataType is used for
varchar-- if that is what you are proposing then I think thestringslt is not enough 🤔Jefffrey commentedon Oct 4, 2025
Resolved by #15096;
varcharshould now beUtf8Viewtype by defaultfindepi commentedon Oct 4, 2025
at dbt, we also had a problem of hard-to-provide test coverage for Utf8, Utf8View.
A
varcharcan be one of them, but the code (eg UDFs) ultimately needs to accept bothwhat we ended up doing is fuzzing in SLT tests. At the test start we generate a random number. Then every string literal is either Utf8, Utf8View or LargeUtf8 -- deterministically based on its value and this random number.
This helped identify many places too rigid or with missing coercion logic (often not only string related), while still allowing us to reproduce any test failures observed on CI.
Jefffrey commentedon Oct 6, 2025
Should we raise a new issue to track this idea? To fuzz not only strings but also lists (largelist, list, fixedsize) and binary/fixedbinary?
alamb commentedon Oct 6, 2025
Yes for sure!
Jefffrey commentedon Oct 7, 2025
Raised #17947 for tracking