Repository navigation
Accelerate row-level MERGE / UPDATE / DELETE plans (MergeRowsExec, ReplaceDataExec, WriteDeltaExec) #5122
Description
What is the problem the feature request solves?
Row-level MERGE / UPDATE / DELETE plans fall back to Spark entirely: MergeRowsExec (Spark 3.5+), ReplaceDataExec, WriteDeltaExec, and InsertOnlyMergeExec (Spark 4.2+).
CDC upsert into Iceberg is one of the most common modern ETL workloads. The expensive parts of a MERGE plan (the source/target join, filters, projections) are operators Comet already accelerates, but the merge tail forces the stage back to Spark, so in practice the whole statement runs without acceleration.
Describe the potential solution
MergeRowsExec itself is close to a projection: it routes each joined row through matched / not-matched instruction lists that are ordinary Catalyst expressions, emitting updated, inserted, or deleted rows plus the row-operation column. That makes it a candidate for a native operator independent of native writes, which would at least keep the join and merge logic in one native stage before handing off to the JVM writer.
Full acceleration of ReplaceDataExec / WriteDeltaExec also needs native DataSource V2 writes, tracked in #5121.
Additional context
Related: #5121 (DataSource V2 writes), #1625 (EPIC: native Parquet writes), #3756 (Iceberg feature matrix).
Activity
docs: link tracking issues for operator acceleration gaps
peterxcli commentedon Jul 31, 2026
take, will start with
MergeRowsExecfirstunikdahal commentedon Aug 7, 2026
Hi @peterxcli
I didn't realise we had an issue for this. I was going through the native iceberg write implementation #4487 and found out that merge into was not supported.
I have a working implementation for MergeRowsExec built currently on top of #4487 . which he's now splitting into parts landing against the merged split-operator work (#4658; #5298 is part 2, part 3 not yet started).
If you're already working on MERGE INTO, that's totally fine, happy to step back. If not, I have a working implementation done and tested on my end for native CometMergeRowsExec, just waiting on part 3 to land before it can target main.
Let me know either way!
peterxcli commentedon Aug 7, 2026
I haven't started to work. Please go ahead. I think you could open draft PR first before the final split of iceberg native patch is landed, please also ping me to review when your PR is ready.
unikdahal commentedon Aug 7, 2026
Thanks @peterxcli, appreciate it! Will polish it up and open a draft PR soon, will tag you for review once it's ready.
andygrove commentedon Sep 22, 2026
Status update, since some of this has landed:
mainReplaceDataExec(copy-on-write DELETE / UPDATE / MERGE)spark.comet.write.iceberg.splitOperator.enabledandspark.comet.iceberg.write.enabledflags. Copy-on-write MERGE still falls back to the JVM write path becauseMergeRowsExecisn't native yet.MergeRowsExecWriteDeltaExec(merge-on-read)WriteDelta.InsertOnlyMergeExec(Spark 4.2+)The dependency on native DataSource V2 writes mentioned above now lives in #5649 (#5121 is closed in its favor), and this issue is tracked there under phase 3. Keeping this open for the merge-on-read and insert-only MERGE paths.