Skip to content

examples: replace sql_dialect with custom_sql_parser example - #19383

Merged
Jefffrey merged 3 commits into
apache:mainfrom
geoffreyclaude:feat/attach
Dec 25, 2025
Merged

Jefffrey merged 3 commits into
apache:mainfrom
geoffreyclaude:feat/attach

Conversation

@geoffreyclaude

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

Related to #16756

Rationale for this change

The existing sql_dialect.rs example demonstrates COPY ... STORED AS ..., which is actually already fully supported by the standard DFParser.

This PR replaces it with the example from #16756: CREATE EXTERNAL CATALOG ... STORED AS ... LOCATION ... with automatic table discovery.

What changes are included in this PR?

The first commit updates dialect.rs to show that DFParser already handles COPY ... STORED AS, making it clear this syntax doesn't need customization.

Example output from cargo run --example sql_ops -- dialect:

Query: COPY source_table TO 'file.fasta' STORED AS FASTA
--- Parsing without extension ---
Standard DFParser: Parsed as Statement::CopyTo: COPY source_table TO file.fasta STORED AS FASTA

--- Parsing with extension ---
Custom MyParser: Parsed as MyStatement::MyCopyTo: COPY source_table TO 'file.fasta' STORED AS FASTA

The second commit adds a new custom_sql_parser.rs example that implements CREATE EXTERNAL CATALOG my_catalog STORED AS <format> LOCATION '<url>' with automatic table discovery from object storage. It also removes the old dialect.rs example.

Are these changes tested?

Yes, the new example is runnable with cargo run --example sql_ops -- custom_sql_parser and demonstrates the full flow from parsing custom DDL through registering the catalog to querying discovered tables.

Example output:

=== Part 1: Standard DataFusion Parser ===

Parsing: CREATE EXTERNAL CATALOG parquet_testing
         STORED AS parquet
         LOCATION 'local://workspace/parquet-testing/data'
         OPTIONS (
           'schema_name' = 'staged_data',
           'format.pruning' = 'true'
         )

Error: SQL error: ParserError("Expected: TABLE, found: CATALOG at Line: 1, Column: 17")

=== Part 2: Custom Parser ===

Parsing: CREATE EXTERNAL CATALOG parquet_testing
         STORED AS parquet
         LOCATION 'local://workspace/parquet-testing/data'
         OPTIONS (
           'schema_name' = 'staged_data',
           'format.pruning' = 'true'
         )

  Target Catalog: parquet_testing
  Data Location: local://workspace/parquet-testing/data
  Resolved Schema: staged_data
  Registered 69 tables into schema: staged_data
Executing: SELECT id, bool_col, tinyint_col FROM parquet_testing.staged_data.alltypes_plain LIMIT 5

+----+----------+-------------+
| id | bool_col | tinyint_col |
+----+----------+-------------+
| 4  | true     | 0           |
| 5  | false    | 1           |
| 6  | true     | 0           |
| 7  | false    | 1           |
| 2  | true     | 0           |
+----+----------+-------------+

Are there any user-facing changes?

Documentation only. I replaced the sql_dialect.rs example with custom_sql_parser.rs and updated the README. No API changes.

Updates the example to show that the 'COPY ... STORED AS' use case is
already supported by the standard DFParser. This baseline shows the
existing example may be confusing as it doesn't highlight a unique
need for a custom dialect, preparing for a more distinct replacement.
@geoffreyclaude geoffreyclaude changed the title feat: replace sql_dialect with custom_sql_parser example examples: replace sql_dialect with custom_sql_parser example Dec 18, 2025
@geoffreyclaude
geoffreyclaude force-pushed the feat/attach branch 6 times, most recently from 56db936 to 8b7794c Compare December 18, 2025 10:20
Demonstrates extending DataFusion's SQL parser to support:
CREATE EXTERNAL CATALOG my_catalog STORED AS <format> LOCATION '<url>'

Implements automatic table discovery and programmatic catalog registration.
Comment thread datafusion-examples/examples/sql_ops/custom_sql_parser.rs
Comment thread datafusion-examples/examples/sql_ops/custom_sql_parser.rs
Comment thread datafusion-examples/examples/sql_ops/custom_sql_parser.rs
Comment thread datafusion-examples/examples/sql_ops/custom_sql_parser.rs Outdated
@Jefffrey
Jefffrey added this pull request to the merge queue Dec 25, 2025
Merged via the queue into apache:main with commit ef2c1a3 Dec 25, 2025
27 checks passed
@Jefffrey

Copy link
Copy Markdown
Contributor

Thanks @geoffreyclaude & @martin-g

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants