schema_force_view_type configuration not working for CREATE EXTERNAL TABLE

Describe the bug

schema_force_view_type configuration said:

datafusion.execution.parquet.schema_force_view_types | true | (reading) If true, parquet reader will read columns of Utf8/Utf8Large with Utf8View, and Binary/BinaryLarge with BinaryView.

Reference: https://datafusion.apache.org/user-guide/configs.html

I tried the following script within datafusion-cli

set datafusion.execution.parquet.schema_force_view_types = true;

CREATE EXTERNAL TABLE IF NOT EXISTS lineitem (
        l_orderkey BIGINT,
        l_partkey BIGINT,
        l_suppkey BIGINT,
        l_linenumber INTEGER,
        l_quantity DECIMAL(15, 2),
        l_extendedprice DECIMAL(15, 2),
        l_discount DECIMAL(15, 2),
        l_tax DECIMAL(15, 2),
        l_returnflag VARCHAR,
        l_linestatus VARCHAR,
        l_shipdate DATE,
        l_commitdate DATE,
        l_receiptdate DATE,
        l_shipinstruct VARCHAR,
        l_shipmode VARCHAR,
        l_comment VARCHAR
) STORED AS parquet
LOCATION '/Users/yongting/Code/datafusion/benchmarks/data/tpch_sf10/lineitem';

select arrow_typeof(l_comment) from lineitem limit 1;

The data type is Utf8 instead Utf8View

+----------------------------------+
| arrow_typeof(lineitem.l_comment) |
+----------------------------------+
| Utf8                             |
+----------------------------------+

To Reproduce

No response

Expected behavior

No response

Additional context

No response

take

@2010YOUY01

Submitted the PR for review.

cc @2010YOUY01 @alamb @xudong963

@zhuqi-lucas

Thank you @zhuqi-lucas

If you change the example slightly (so the column names are not explicitly listed) then the type is correctly set to Utf8View

> CREATE EXTERNAL TABLE IF NOT EXISTS lineitem STORED AS parquet
LOCATION '/Users/andrewlamb/Software/datafusion/benchmarks/data/tpch_sf10/lineitem';
0 row(s) fetched.
Elapsed 0.010 seconds.

> select arrow_typeof(l_comment) from lineitem limit 1;
+----------------------------------+
| arrow_typeof(lineitem.l_comment) |
+----------------------------------+
| Utf8View                         |
+----------------------------------+
1 row(s) fetched.
Elapsed 0.015 seconds.

I think what is happening is that the SQL type VARCHAR is mapped to Utf8

datafusion/datafusion/sql/src/planner.rs

Line 561 in c0d53ad

_ => Ok(DataType::Utf8),

So when the table is created with an explicit type it is set to Utf8

However, when the types are read directly from Parquet the types is inferred as Utf8View

@zhuqi-lucas

Huge kudos to @zhuqi-lucas for trying to fix this in

BUG: schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14922

After some research, I think we have concluded the real reason this isn't working is

Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096

If we need a shorter term workaround for this problem, I suggest we start with a config settings:

Change mapping of SQL VARCHAR from Utf8 to Utf8View #15096 (comment)

Navigation Menu

Repository navigation

`schema_force_view_type` configuration not working for `CREATE EXTERNAL TABLE` #14909

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

schema_force_view_type configuration not working for CREATE EXTERNAL TABLE #14909

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

zhuqi-lucas commented on Feb 27, 2025

zhuqi-lucas commented on Mar 3, 2025

alamb commented on Mar 5, 2025

alamb commented on Mar 8, 2025

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`schema_force_view_type` configuration not working for `CREATE EXTERNAL TABLE` #14909