Skip to content

Support AVRO Format for Write Queries #7679

Closed
#24925
Closed
Support AVRO Format for Write Queries#7679
#24925
Assignees
jecsand838
Labels
enhancementNew feature or request

Description

@devinjdangelo
devinjdangelo
opened on Sep 28, 2023

Is your feature request related to a problem or challenge?

Writing Avro files is currently not supported in DataFusion, such as via COPYor INSERT INTO queries. This would be a nice feature to have and recently came up in the discord channel.

Describe the solution you'd like

  • Implement create_write_physical_plan for AvroFormat
  • Implement FileSink on a AvroSink struct
  • Implement file writing options / string parsing in FileTypeWriterOptions for avro

I expect some of the common functions in https://github.com/apache/arrow-datafusion/blob/main/datafusion/core/src/datasource/file_format/write.rs can be reused in the avro implementation.

Describe alternatives you've considered

Don't support writing avro files

Additional context

No response

Activity

  1. devinjdangelo
    added
    enhancementNew feature or request
    on Sep 28, 2023

  2. Veeupup commented on Dec 4, 2023

    @Veeupup
    Veeupup
    on Dec 4, 2023
    Contributor

    @alamb Hi, I can help with this ticket : )

  3. alamb commented on Dec 4, 2023

    @alamb
    alamb
    on Dec 4, 2023
    Contributor

    @alamb Hi, I can help with this ticket : )

    Thanks @Veeupup -- for this one the first thing we need to do is get an avro writer and the second part is hooking it into DataFusion. There is some work upstream in arrow-rs to make an avro reader/writer in apache/arrow-rs#4886 but I think that project is stilled at the moment

  4. jecsand838 commented on Oct 16, 2025

    @jecsand838
    jecsand838
    on Oct 16, 2025
    Contributor

    @alamb @devinjdangelo

    Since apache/arrow-rs#4886 is now closed and arrow-avro is set for release with arrow-rs v57.0.0 this month, we could probably resume progress here.

    I wouldn't mind taking this and having it ready for the DataFusion v51 release.

  5. alamb commented on Oct 16, 2025

    @alamb
    alamb
    on Oct 16, 2025
    Contributor

    In case you want to start early, I have a PR that integrates with the latest arrow main that you could use until we actually release arrow 57 (likely will be late next week)

  6. jecsand838 commented on Oct 20, 2025

    @jecsand838
    jecsand838
    on Oct 20, 2025
    Contributor

    take

  7. github-actions
    assigned
    jecsand838
    on Oct 20, 2025

  8. pepijnve commented on Sep 3, 2026

    @pepijnve
    pepijnve
    on Sep 3, 2026
    Contributor

    There hasn't been much activity for a while related to this issue, so I've created a draft PR that adds the necessary wiring. Still in draft because the code was AI authored and I still want to go through it one more time before marking it ready for review.

  9. pepijnve
    added a commit that references this issue on Sep 3, 2026

    apache#7679 Refine error mapping

    135c60b
  10. kosiew
    closed this as completedin #24925on Sep 10, 2026
  11. rluvaton
    added a commit that references this issue on Sep 10, 2026

    Add write support to Avro format (apache#24925)

    a7ea120
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

enhancementNew feature or request

Type

No type

Projects

No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions