Skip to content

Tracking issues of aligning storage support with iceberg-java #408

Closed as not planned
Closed as not planned
Tracking issues of aligning storage support with iceberg-java#408
Assignees
Xuanwo
Labels
good first issueGood for newcomershelp wantedExtra attention is neededstale

Description

@Xuanwo
Xuanwo
opened on Jun 19, 2024

iceberg-java now supports

  • aws s3
  • aliyun oss
  • azure adlsv2
  • gcp gcs
  • hadoop hdfs

Although OpenDAL supports more storage services than this, it still makes sense to at least support all existing storage services. This issue will track the progress.

After this been implemented, iceberg-rust will have the same storage support level as iceberg-java. I'm willing implement those features but also open to help review related changes. Please comment if you want to join the development and pick up one of them.

Activity

  1. Xuanwo
    self-assigned this
    on Jun 19, 2024

  2. Xuanwo commented on Jun 19, 2024

    @Xuanwo
    Xuanwo
    on Jun 19, 2024
    MemberAuthor

    cc @liurenjie1024, I'm not sure about your 0.3 release plan going. Maybe we can include this one inside?

    Most changes should be easy and no API changes. It's also fine to be included in the following 0.3.x releases.

  3. Xuanwo
    added
    good first issueGood for newcomers
    help wantedExtra attention is needed
    on Jun 19, 2024

  4. liurenjie1024 commented on Jun 19, 2024

    @liurenjie1024
    liurenjie1024
    on Jun 19, 2024
    Contributor

    Hi, @Xuanwo There are two places to track 0.3 features:

    1. Tracking issues of iceberg-rust v0.3.0 #348
    2. https://github.com/apache/iceberg-rust/milestone/2

    I'm ok with waiting for adding this into 0.3 release. I'm just curious how to test against these? Or maybe we can start with declaring these features as experimental.

  5. liurenjie1024
    added this to iceberg-ruston Jun 19, 2024
  6. liurenjie1024
    added this to the 0.3.0 Release milestone on Jun 19, 2024

  7. Xuanwo commented on Jun 19, 2024

    @Xuanwo
    Xuanwo
    on Jun 19, 2024
    MemberAuthor

    I'm just curious how to test against these? Or maybe we can start with declaring these features as experimental.

    • aws s3: tested by minio now. We can add real s3 bucket in with sponsor.
    • aliyun oss: need an oss bucket (better to locate near us-east-1)
    • azure adlsv2: can be tested by Azurite. And I'm willing to provide test infra as Microsoft MVP.
    • gcp gcs: need a gcs bucket (better to locate near us-east-1)
    • hadoop hdfs: can setup in CI directly (thanks open source!)

    I agree that we can label these features as experimental. Setting up the CI infrastructure requires time, more so than implementing those features.

  8. sdd commented on Jun 19, 2024

    @sdd
    sdd
    on Jun 19, 2024
    Contributor

    I have a question on the current FileIO - @Xuanwo is probably the right person to ask here.

    It would be useful to be able to customize the OpenDAL Operator by being able to attach layers. Could we extend expose this capability somewhere? I've more than happy to work on this.

  9. liurenjie1024 commented on Jun 19, 2024

    @liurenjie1024
    liurenjie1024
    on Jun 19, 2024
    Contributor

    I'm just curious how to test against these? Or maybe we can start with declaring these features as experimental.

    • aws s3: tested by minio now. We can add real s3 bucket in with sponsor.
    • aliyun oss: need an oss bucket (better to locate near us-east-1)
    • azure adlsv2: can be tested by Azurite. And I'm willing to provide test infra as Microsoft MVP.
    • gcp gcs: need a gcs bucket (better to locate near us-east-1)
    • hadoop hdfs: can setup in CI directly (thanks open source!)

    I agree that we can label these features as experimental. Setting up the CI infrastructure requires time, more so than implementing those features.

    Cool, let's move!

  10. Xuanwo commented on Jun 19, 2024

    @Xuanwo
    Xuanwo
    on Jun 19, 2024
    MemberAuthor

    It would be useful to be able to customize the OpenDAL Operator by being able to attach layers. Could we extend expose this capability somewhere? I've more than happy to work on this.

    Any detailed ideas? Are you talking about enabling some existing layers for opendal or allow users to implement something new based on FileIO?

    I can imagine that enabling logging and retry layers by default or by configuring might be useful.

  11. Xuanwo commented on Jun 19, 2024

    @Xuanwo
    Xuanwo
    on Jun 19, 2024
    MemberAuthor

    I agree that we can label these features as experimental. Setting up the CI infrastructure requires time, more so than implementing those features.

    Split into a new issue: #410.

    I plan to track them after 0.3 release.

  12. jsimbadev commented on Jul 4, 2024

    @jsimbadev
    jsimbadev
    on Jul 4, 2024

    @Xuanwo I can take the Azure datalake FileIO Implementation + the corresponding infrastructure set up, sound ok?

  13. Xuanwo commented on Jul 5, 2024

    @Xuanwo
    Xuanwo
    on Jul 5, 2024
    MemberAuthor

    @Xuanwo I can take the Azure datalake FileIO Implementation + the corresponding infrastructure set up, sound ok?

    Welcome, have fun!

  14. liurenjie1024 commented on Aug 6, 2024

    @liurenjie1024
    liurenjie1024
    on Aug 6, 2024
    Contributor

    cc @Xuanwo Do you still plan to finish this before in 0.3.0? Or we can postpone it to next release?

  15. Xuanwo commented on Aug 6, 2024

    @Xuanwo
    Xuanwo
    on Aug 6, 2024
    MemberAuthor

    cc @Xuanwo Do you still plan to finish this before in 0.3.0? Or we can postpone it to next release?

    There are some more work to do at opendal side. I believe we can let 0.3.0 go first.

  16. liurenjie1024
    removed this from the 0.3.0 Release milestone on Aug 6, 2024

  17. github-actions commented on Oct 9, 2025

    @github-actions
    github-actionsBot
    on Oct 9, 2025 – with GitHub Actions
    Contributor

    This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

  18. github-actions
    added
    stale
    on Oct 9, 2025

  19. github-actions commented on Oct 24, 2025

    @github-actions
    github-actionsBot
    on Oct 24, 2025 – with GitHub Actions
    Contributor

    This issue has been closed because it has not received any activity in the last 14 days since being marked as 'stale'

  20. github-actions
    closed this as not plannedon Oct 24, 2025
  21. github-project-automation
    moved this to Done in iceberg-ruston Oct 24, 2025
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

good first issueGood for newcomershelp wantedExtra attention is neededstale

Type

No type

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions