Skip to content

Remove DFParser #4808

Closed
Closed
Remove DFParser#4808
Assignees
Jefffrey
Labels
enhancementNew feature or request

Description

@tustvold
tustvold
opened on Jan 3, 2023

Is your feature request related to a problem or challenge? Please describe what you are trying to do.

DataFusion contains a custom DFParser that extends sqlparser and overrides its parsing of DESCRIBE TABLES and CREATE EXTERNAL TABLE.

Describe the solution you'd like

Looking at sqlparser the only functionality missing upstream is understanding the COMPRESSION keyword in CREATE EXTERNAL TABLE. This feels like a very simple upstream contribution that would then allow removing DFParser and its corresponding Statement abstraction.

Describe alternatives you've considered

We could not do this

Additional context

FYI @andygrove @alamb @liukun4515

Activity

  1. tustvold
    added
    enhancementNew feature or request
    on Jan 3, 2023
  2. tustvold
    closed this as completedon Jan 3, 2023

  3. tustvold commented on Jan 3, 2023

    @tustvold
    tustvold
    on Jan 3, 2023
    ContributorAuthor

    Actually I think I would like to use Statement, to remove the DDL from LogicalPlan

  4. andygrove commented on Jan 3, 2023

    @andygrove
    andygrove
    on Jan 3, 2023
    Member

    Makes sense to me.

    Also, just as an FYI, when DFParser was introduced, we did not have a good mechanism for selectively overriding parsing in sqlparser-rs but we do now. It should now be possible for us to implement a DataFusion Dialect and override parsing at the statement, prefix, and infix levels. See tests at https://github.com/sqlparser-rs/sqlparser-rs/blob/main/tests/sqlparser_custom_dialect.rs for examples. I'm not sure the impact of this approach would be on supporting multiple dialects though.

  5. tustvold
    reopened this on Jan 3, 2023

  6. unconsolable commented on Jan 5, 2023

    @unconsolable
    unconsolable
    on Jan 5, 2023
    Contributor

    There are some differences in datafusion's CreateExternalTable and sqlparser's CreateTable, for example has_header , delimiter, table_partition_cols and file_compression_type are not exist in sqlparser's CreateTable. Are there any way to create custom Statement in sqlparser?

  7. tustvold commented on Jan 5, 2023

    @tustvold
    tustvold
    on Jan 5, 2023
    ContributorAuthor

    The partitoning is under one of the Hive options, not sure about the others. I believe there is support for custom dialects, but in this case I'd vote for upstreaming anything missing - we don't have any especially esoteric needs AFAIK

  8. Jefffrey commented on Jan 1, 2024

    @Jefffrey
    Jefffrey
    on Jan 1, 2024
    Contributor

    I'd like to take this on. I'll try do the migration in steps to make PRs more digestible and ensure no breakages/regressions.

    First step is here: #8703

  9. alamb
    assigned
    Jefffrey
    on Jan 1, 2024

  10. alamb commented on Jan 2, 2024

    @alamb
    alamb
    on Jan 2, 2024
    Contributor

    Related conversation in apache/datafusion-sqlparser-rs#1080

    From what I know, the CREATE EXTERNAL TABLE syntax is not based on any other dialect and thus is DataFusion specific. I think it may not belong upstream in sqlparser-rs

    The COPY syntax I tried to base on DuckDB's copy syntax (which is different than Postgres), so if we added DuckDB style COPY support to sqlparser-rs we could probably migrate DataFusion to use it

  11. alamb commented on Feb 3, 2025

    @alamb
    alamb
    on Feb 3, 2025
    Contributor

    I don't think we plan to do this / remove the EXTERNAL TABLE syntax so let's close this ticket to avoid confusion

  12. alamb
    closed this as completedon Feb 3, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions