HDFS support in iceberg-rust

There are several ways to access HDFS in rust:

libhdfs + JNI: needs java runtime and libhdfs
WebHDFS: http based, may have performance issues
HDFS Native: rust native but not well adopted.

OpenDAL supports all these methods. However, from Iceberg's perspective, it's better to expose a single hdfs storage option and allow users to choose the method via feature flags to avoid confusion. This is because we write service names in our metadata files, and all implementations must be able to read hdfs://host:port/files.

So my current plan is to introduce webhdfs and hdfs_native support in iceberg-rust. They will hide under storage-webhdfs and storage-hdfs-native flag:

Users can choose which implementations to use:

fn parse_scheme(scheme: &str) -> crate::Result<Scheme> {
    match scheme {
        "memory" => Ok(Scheme::Memory),
        "file" | "" => Ok(Scheme::Fs),
        "s3" | "s3a" => Ok(Scheme::S3),
        "gs" | "gcs" => Ok(Scheme::Gcs),
        #[cfg(feature = "storage-webhdfs")]
        "hdfs" => Ok(Scheme::Webhdfs),
        #[cfg(feature = "storage-hdfs-native")]
        "hdfs" => Ok(Scheme::HdfsNative),
        s => Ok(s.parse::<Scheme>()?),
    }
}

See #1131 for more details about this design.

@liurenjie1024

cc @liurenjie1024, let's discuss here.

I'm also fine to add a new services called webhdfs here. But the location returned by hive will still be hdfs://host:port/abc. It will be a bit confusing.

cc @liurenjie1024, let's discuss here.

I'm also fine to add a new services called webhdfs here. But the location returned by hive will still be hdfs://host:port/abc. It will be a bit confusing.

Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?

Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?

Yep. There is no iceberg impls will write webhdfs://xxx.

Do you mean that even if user is using webhdfs, hive metastore will still return hdfs path?

Yep. There is no iceberg impls will write webhdfs://xxx.

Seems there are several problems needs to resolve here:

Should we add webhdfs support as feature? It's mainly compile time support or runtime support for webhdfs
Should we treat webhdfs/hdfs as same storage or different service?

Is there other problems to discuss?

It's mainly compile time support or runtime support for webhdfs

What's the difference about this?

It's mainly compile time support or runtime support for webhdfs

What's the difference about this?

Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.

Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.

I see, you mean that we have a single Storage Hdfs but provides a config like use_webhdfs for users to switch?

Oh, I was thinking about treating hdfs/webhdfs as same storage service. I took a look your pr, if they are treated as different stoarge, then there is difference.

I see, you mean that we have a single Storage Hdfs but provides a config like use_webhdfs for users to switch?

Yes, but I'm not sure it's the best approach.

I see. We have three approachs so far:

Support storage-webhdfs and storage-hdfs but only exposes as hdfs to users

Users are required to use feature flags for configuration.

Have only storage-hdfs but allow users to pick up impls like webhdfs or hdfs at runtime by config.

Provides more control for users, but it may be a bit confusing when feature flags don't match the configuration.

Exposes storage-webhdfs and storage-hdfs seperately

It doesn't work for webhdfs since all tables will use hdfs://abc.

Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?

I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.

Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?

Yep, I intentionally leaving it out for now since I don't want to play with java runtime for now. And I'm guessing adding it doesn't need another approach. We can just follow the same way to handle it.

I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.

Oh, that's something new to me. We just need tp map viewfs to hdfs, right?

Have you considered including libhdfs + JNI in this design? Are we intentionally leaving it out for now, or could it be explored as a separate approach?

Yep, I intentionally leaving it out for now since I don't want to play with java runtime for now. And I'm guessing adding it doesn't need another approach. We can just follow the same way to handle it.

I actually tested accessing HDFS using libhdfs + JNI myself, and it worked! The main issue I ran into was linking the JVM library.
It’d be awesome if support for this could be added here eventually.

I was wondering if the viewfs:// scheme might be supported in this context? I’ve tested it with OpenDAL and confirmed that it’s already supported there.

Oh, that's something new to me. We just need tp map viewfs to hdfs, right?

To clarify, I ran my test using libhdfs + JNI, and I’m not entirely sure if viewfs is supported in the other two approaches.
In my experience with libhdfs + JNI, viewfs is handled the same way as hdfs, with libhdfs taking care of the differences under the hood. That said, it seems like we’d only need to explicitly pass the scheme to OpenDAL for it to work smoothly.

Sorry that I had to leave early today at the sync, it was a great discussion. Both in Java and PyIceberg there is a FileIO property that you can set to force a specific FileIO: https://py.iceberg.apache.org/configuration/#fileio. This is slightly different since all the implementations come from OpenDAL, but might be an option to consider.

Per yesterday's discussion, we reached consensus on following points:

We should treat webhdfs as a storage different hdfs given their configuration would be different.
We should include both libhdfs and hdfsrs in iceberg-rust and allow user to choose at runtime. Libhdfs is still much more mature than hdfsrs so in serious hadoop deployment it's still useful. Hdfsrs allows user to run without relying on, so we think it's still valueful in some use cases.

AFAIK, webhdfs doesn't support HDFS NameNode HA. When fail over happens, it won't switch to the new active NameNode automatically.

Does iceberg rust support hadoop catalog?
I only see s3tables in the code base.

@mingmwang

Does iceberg rust support hadoop catalog? I only see s3tables in the code base.

Hi, @mingmwang There is no support for hadoop catalog, but there is StaticTable which allows you to query tables without relying on a catalog.

@Xuanwo

@Xuanwo Are you doing some experiment with AI summary?

OpenDAL的HDFS写路径问题挺多的

而且OpenDAL套路有点深，问题很难triage

@Xuanwo Are you doing some experiment with AI summary?

Oh, sorry about that. My AI summary bot accidentally posted the comments publicly.

Any plans to resurrect this feature? Hadoop is still a huge thing and it's confusing we can't access Iceberg tables backed by HDFS declaring it's supported in docs

Any plans to resurrect this feature? Hadoop is still a huge thing and it's confusing we can't access Iceberg tables backed by HDFS declaring it's supported in docs

About hdfs there are two things:

FileIO support for hdfs. I think this is in the roadmap.
Catalog support for hdfs. This is out of scope and deprecated by community.

This issue has been automatically marked as stale because it has been open for 180 days with no activity. It will be closed in next 14 days if no further activity occurs. To permanently prevent this issue from being considered stale, add the label 'not-stale', but commenting on the issue is preferred when possible.

@liurenjie1024

@liurenjie1024
"Hadoop is still a huge thing " ... still huge and used widely
This is main blocker in datafusion-comet to read Natively HDFS based Iceberg tables.

Don't remove this.

refactor(storage): rename hdfs module to hdfs_native, drop unused Hdf…

Navigation Menu

Repository navigation

HDFS support in iceberg-rust #1130

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

HDFS support in iceberg-rust #1130

Description

Activity

Xuanwo commented on Mar 25, 2025

liurenjie1024 commented on Mar 25, 2025

Xuanwo commented on Mar 25, 2025

liurenjie1024 commented on Mar 25, 2025

Xuanwo commented on Mar 25, 2025

liurenjie1024 commented on Mar 25, 2025

Xuanwo commented on Mar 25, 2025

liurenjie1024 commented on Mar 25, 2025

Xuanwo commented on Mar 25, 2025

oven-yang commented on Mar 27, 2025

oven-yang commented on Mar 27, 2025

Xuanwo commented on Mar 27, 2025

oven-yang commented on Mar 27, 2025

Fokko commented on Mar 27, 2025

liurenjie1024 commented on Mar 28, 2025

manuzhang commented on Mar 28, 2025

mingmwang commented on Apr 11, 2025

liurenjie1024 commented on Apr 11, 2025

Xuanwo commented on Sep 12, 2025

manuzhang commented on Sep 12, 2025

mingmwang commented on Sep 12, 2025

mingmwang commented on Sep 12, 2025

Xuanwo commented on Sep 12, 2025

EugeneYushin commented on Nov 19, 2025

liurenjie1024 commented on Nov 20, 2025

github-actions commented on May 20, 2026

mixermt commented on Aug 10, 2026

blackmwk commented on Aug 10, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions