Copy-on-Write and Merge-on-Read support #2186

Activity

1. glitchy

   added

   epicEpic issueEpic issue

   on Feb 27, 2026
2. wirybeaver commented on Mar 3, 2026

   

   wirybeaver

   on Mar 3, 2026

   Last edited by wirybeaver

   More actions

   Hi @glitchy, I am planning to contribute the merge_into for the datafusion iceberg-rust. The implementation requires to the fundanmental RowDeltaAction. I will implement the merge_into and RowDeltaAction for CoW at first. #2201
3. glitchy commented on Mar 3, 2026

   

   glitchy

   on Mar 3, 2026

   Last edited by glitchy

   Author

   More actions

   hey @wirybeaver --thanks for the interest in this area! a few things worth flagging:

   RowDeltaAction overlaps with OverwriteAction

   the OverwriteAction in my PR (#2185) already handles adding data files and marking existing ones as deleted via manifest rewriting--add_data_files(), delete_data_files(), Operation::Overwrite, ManifestStatus::Deleted entries with preserved sequence numbers. looking at your RowDeltaAction (#2201), it reimplements the same core logic. RowDeltaAction is already planned for future PRs in this series (see checklist above)

   datafusion is a consumer, not a dependency

   the core Iceberg transaction primitives (OverwriteAction, RowDeltaAction, scan-side delete reconciliation) do not depend on datafusion--they live in the iceberg crate and need to work independently of any query engine. datafusion integration (INSERT OVERWRITE, MERGE INTO) is a layer on top that consumes these APIs. coupling them together makes both harder to review, and datafusion doesn't even support MERGE INTO parsing yet (there's no logical plan node for it and it falls through to not_impl_err!("Unsupported SQL statement") in their planner).

   the sequencing for the core primitives is already outlined in the checklist above on this issue. once they go in then datafusion integration should be considered.
4. wirybeaver commented on Mar 4, 2026

   

   wirybeaver

   on Mar 4, 2026

   Last edited by wirybeaver

   More actions

   Hi @glitchy , thanks for the context!

   To keep the momentum going for both merge_into and the update DataFusion integration, I’d suggest we decouple the basic RowDeltaAction (CoW) from the more complex MoR implementation.

   Since I have the CoW implementation ready and tested it together with for my merge_into full PoC #2201 and update full PoC #2205, merging the "bare" RowDeltaAction now would:

   • Unblock downstream PRs for merge_into and update immediately.
   • Reduce the scope of your MoR PR, making it easier and faster for maintainers to review.

   I’m happy to rebase on your OverwriteAction as soon as it lands to ensure zero conflicts. This way, we aren't serializing the work—I can push the CoW foundation through now, and you can layer the complex MoR logic on top of it without having to worry about the boilerplate.

   In terms of datafusion support on merge_into, I already noticed this unsupported problem, mentioned this in my epic note and plan to contribute a merge_into on the datafusion side. We can first implement a merge_into TableProvider hook like this PR do on the datafusion side: apache/datafusion#19142

   Does that sound like a fair way to get both features landed faster?
5. malon64 commented on Apr 20, 2026

   

   malon64

   on Apr 20, 2026

   More actions

   Hi, I’m following this work from the perspective of a downstream Rust data ingestion tool.

   From our side, the first valuable milestone would already be a clean CoW-oriented primitive that can support SCD1-style keyed upserts safely, even before full SQL MERGE ergonomics are available.

   Happy to help by:

   • testing proposed APIs/branches from a real downstream use case,
   • sharing concrete acceptance scenarios for keyed upsert behavior,
   • and possibly contributing on smaller enabling pieces if useful.
6. glitchy commented on Jun 13, 2026

   

   glitchy

   on Jun 13, 2026

   Author

   More actions

   Thanks for your support, @malon64 --unfortunately I had to step away from this project, but am now back to continue the push.

   My first two PRs will deliver your CoW-oriented primitive:

   • OverwriteAction (feat(transaction): add OverwriteAction with CoW delete support #2185) is the CoW primitive
   • The following PR will introduce validateNoConflictingData / validateNoConflictingDeletes which will allow for safe upserts under concurrency

   Feel free to contribute and all comments are most welcome. Will ping you when the second PR lands--thank you!

   👍React with 👍1Reacted by Alexis
7. glitchy

   added 2 commits that reference this issue on Jun 30, 2026

   feat(transaction): add validateNoConflictingData for overwrite

   ...

   751f067

   feat(transaction): add validateNoConflictingData for overwrite

   ...

   b42461e
8. ScottSyms

   added a commit that references this issue on Aug 18, 2026

   feat: idempotent Iceberg commits via per-partition object manifest

   ...

   de8ee74