Skip to content

Support multiple_members on compression decoders? #9156

Closed

Description

@tshauck

Is your feature request related to a problem or challenge?

The challenge is that currently when trying to use the gzip decoder from async-compression to decode block-gzipped files, the query ends after the first block in the file not the EOF.

Based on an issue filed here: Nullus157/async-compression#260, bgzip works if the decoder is modified to support multiple members (I've confirmed this is the case).

Describe the solution you'd like

So my feature request is the ability to specify to call this method or have it set by default (though I'm not sure the performance implications).

I'd be happy to take a look a this, but I'm not sure what an appropriate path would be that's congruent with datafusion overall.

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. alamb commented on Feb 8, 2024

    @alamb
    Contributor

    It sounds like the request here is "support block-gzipped files"?

    If so that seems reasonable to me. Maybe we could just add CompressionType::BGZIP or something?

    Edit: now I see #9160 👍

  2. tshauck commented on Feb 8, 2024

    @tshauck
    ContributorAuthor

    Sorry, I did the ol' "ask for the solution I think I want vs what I want in the end"... thought more about it when I started the implementation to open #9160.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions