Repository navigation
Future of Iceberg Support in Comet #2921
Description
What is the problem the feature request solves?
Comet currently has two different approaches to scanning Iceberg tables. One approach is based on integrating with the Iceberg Java library, and the other is based on integrating with the Rust library. Both of these approaches are experimental.
We are looking for feedback from the community on whether to continue with both approaches, or standardize on one approach.
Implementation # 1 - Integrate with Iceberg Java library
Comet contributors have been contributing to the main Iceberg repository to integrate Comet as an optional reader.
One of the major benefits of this approach is that Comet can support some Spark features more easily through JVM code, such as modular encryption and custom authentication.
There have been numerous challenges due to the fact that Comet and Iceberg depend on different versions of various libraries (including Parquet) and this has led to some workarounds involving shading and redesigning the API between Iceberg and Comet. This also adds a circular dependency between the Comet and Iceberg projects, where we need to release new Comet versions before we can update Iceberg to use that version. There have also been challenges in getting PRs merged.
It is currently necessary to patch the Iceberg source and build custom releases.
Implementation # 2 - Integrate with Iceberg Rust library
As of version 0.12.0, Comet now includes an integration with the iceberg-rust crate (which uses the same arrow-rs Parquet reader that Comet already uses). This new approach allows unmodified Iceberg Java to handle query planning (i.e., catalog access, partition pruning, etc.), then Comet serializes Iceberg FileScanTask objects directly to iceberg-rust, enabling native execution of Iceberg table scans through DataFusion.
Parquet Modular Encryption: Encryption was recently added to the Iceberg spec in v3. PME is already supported in Arrow-rs Parquet reader used by Comet. As part of supporting the Iceberg v3 spec, Comet would need to set encryption properties in ArrowReaderOptions that iceberg-rust does not currently expose. We would need to expose the relevant fields in the ArrowReader in a way that is spec compliant.
Describe the potential solution
No response
Additional context
No response
Activity
mbutrovich commentedon Dec 16, 2025
Thanks for getting the conversation started, @andygrove!
I made my initial argument in #2528, particularly in "Benefits over iceberg_compat". While there are execution/runtime/behavior differences between the two code paths, I think our challenges with shading and circular dependencies with Iceberg Java forced us to try the Iceberg Rust approach. It's been tough to make significant progress enhancing the Iceberg Java + Comet code path with new features and bug fixes, not for lack of trying from @parthchandra and @huaxingao.
I believe our ability to rapidly iterate in both the Comet and Iceberg Rust repos (shouts out to @Xuanwo, @kevinjqliu, and others) has been promising so far. Iceberg Rust is about to cut their 0.8 release with significant enhancements to the reader from the Comet team. There are definitely feature gaps to work on, but I feel like we can close them much more quickly with the Iceberg Rust approach.
My opinion is that we should deprecate the Iceberg Java + Comet hybrid scan approach, and focus on hardening and enhancing the Iceberg Rust code path.
coderfender commentedon Dec 16, 2025
Apart from the above mentioned technical advantages, having all core logic in Rust (Implementation #2) might be beneficial in the long run from an OSS / developer community standpoint.
parthchandra commentedon Dec 16, 2025
Adding a clarification -
We have three implementations with Iceberg -
native_comet- Does not support complex types. Support merged into Iceberg code base but is not complete (Spark: Encapsulate parquet objects for Comet iceberg#13786, feat: support deleted rows in Comet iceberg#14062, Spark: Enable Comet Scan in Iceberg iceberg#13793.)native_iceberg_compat- supports complex types. Requires (1). Fails some iceberg tests (involving deleted rows). Still in development (stalled)iceberg_native- Supports complex types. Passes all iceberg tests. Available, but must be enabled explicitly.parthchandra commentedon Dec 16, 2025
We also have three implementations of Parquet scans which correspond to the approaches above-
native_comet- no complex typesnative_iceberg_compat- currently the default choicenative_datafusion- must be enabled explicitly. Needs a little bit of work to pass all tests.manuzhang commentedon Dec 17, 2025
Thanks @andygrove for starting this thread. Can we also bring the discussion to the iceberg community, esp. dev list?
CTTY commentedon Dec 17, 2025
I’m leaning toward prioritizing the iceberg-rust path to avoid the shading and circular dependency headaches of the Java approach. Easier maintenance typically lowers the entry barrier and encourages more community involvement, which is important for the project’s long-term health.
It also makes sense to keep the Iceberg Java integration available as an experimental option for users who need its richer JVM-specific features, while Rust becomes the long-term default. We don’t want to block contributions from anyone who truly needs the Java path.
parthchandra commentedon Dec 17, 2025
Sadly, the Java version is currently broken because of the shading issues.
wForget commentedon Dec 18, 2025
I also prefer the iceberg-rust solution. Additionally, there are discussions about icebeg integration in the auron community, so I think we could work together to improve icebeg-rust. cc @slfan1989 @guixiaowen @richox
Shekharrajak commentedon Dec 20, 2025
For tracking :
liukun4515 commentedon Dec 22, 2025
could you please share the discussion from auron community? @wForget
kevinjqliu commentedon Dec 23, 2025
Thanks for starting this discussion, Andy!
A few initial thoughts,
Implementation # 1 - Integrate with Iceberg Java library
From what I’ve gathered, apache/iceberg#13786 might help resolve this issue. I’ll take a look at the PR.
This is an interesting problem. @pvary has been working on modularizing the different readers in the java iceberg repo, with the File Format API. Perhaps that can help untangle the dependency issue.
Similar to above, happy to help here. If it’s useful to coordinate, perhaps we can start a #comet channel on the Iceberg slack .
For “Implementation # 2 - Integrate with Iceberg Rust library
Thanks to the awesome work from the Comet community (@mbutrovich and perhaps others). Matt reported that he was able to run all of Iceberg Java’s spark tests successfully using the new iceberg-rust 0.8.0 release candidate, an amazing milestone!
Ecosystem
Overall, I feel like iceberg-rust is the ideal place for integration, esp with the datafusion ecosystem. There are already a number of efforts to improve the iceberg-rust <> datafusion integration, which benefits Comet, pyiceberg and likely other projects. Improvements to the iceberg-rust repo can be helpful to the broader data community as a whole.
wForget commentedon Dec 24, 2025
@liukun4515 There hasn’t been a detailed proposal in the Auron community yet, but there have been some related discussions apache/auron#1472, apache/auron#1676, apache/auron#1460. Additionally, as far as I know, @slfan1989 is currently experimenting with this area.
andygrove commentedon Jan 22, 2026
Thanks for all the feedback so far!
I wanted to share a couple of updates in Comet related to this.
I have two PRs in Comet that I am looking for reviews on.
The first PR adds an
@IcebergApiannotation to all Java classes/methods/fields that are currently referenced from the Iceberg main branch, and also adds documentation to the contributors guide.#3237
The second PR builds on this and adds a new
iceberg-apimaven module containing dedicated unit tests for API stability. This PR is more consequential because it discusses deprecating/removing Comet'snative_cometscan and how that relates to the Iceberg public API.#3240
I am not an expert on the current integration, so it is possible that I may have misunderstood things. I would appreciate reviews from both the Iceberg and Comet communities!
andygrove commentedon Mar 12, 2026
The Comet contributors have decided to focus on the
iceberg-rustintegration and stop developing the integration with the Java implementation of Iceberg. We will follow up with PRs against Iceberg over the next few weeks to remove the existing Comet integration, which is broken.manuzhang commentedon Mar 12, 2026
Do you mean we will not continue on apache/iceberg#13786?
andygrove commentedon Mar 12, 2026
Just to clarify, all decisions on code changes in Comet and/or Iceberg repos will be via GitHub issues and PRs, and therefore all decisions actually impacting code will be made "on the mailing list" with community involvement.
manuzhang commentedon Mar 17, 2026
Thanks @andygrove, could you please cross post the decision to the previous discussion on the Iceberg dev list?
andygrove commentedon Mar 18, 2026
PR to remove Comet integatrion from Iceberg (Java):
apache/iceberg#15674