Skip to content

Add example for writing a FileFormat #11079

Description

@alamb

Is your feature request related to a problem or challenge?

Now that @devinjdangelo has added better support for user defined file formats (see #11060)

We have an example of a custom table provider: https://datafusion.apache.org/library-user-guide/custom-table-providers.html and example

It would be great to have an example of creating a file format

Such an example would make sure the APIs added in this PR can be used for the end to end case, as well as obviously made it easier for others to discover and use this feature

Describe the solution you'd like

Add an example in https://github.com/apache/datafusion/tree/main/datafusion-examples that does:

  1. Creates a new file format (something basic like use CSV / ROT13)
  2. reads / writes data using that format

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. tshauck commented on Jun 23, 2024

    @tshauck
    Contributor

    take

  2. tshauck commented on Jun 23, 2024

    @tshauck
    Contributor

    I'll take this as I want to try it out. I do this for one of my libraries via the parser plus user defined nodes, so I want to see what this looks like as an alternative.

  3. alamb commented on Jun 23, 2024

    @alamb
    ContributorAuthor

    I'll take this as I want to try it out. I do this for one of my libraries via the parser plus user defined nodes, so I want to see what this looks like as an alternative.

    Thanks @tshauck -- note #11060 isn't merged yet, so you may have to start the example against that branch rather than main

  4. alamb commented on Jun 28, 2024

    @alamb
    ContributorAuthor

    #11060 is now merged 🎉

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions