[kinolaev]

Equality delete files can be referenced by many data files. This PR allows to accumulate all equality deletes of a table snapshot in the structure spec id -> partition -> equality field ids -> equality key -> max data sequence number that can be reused for many data files. If passed to DeleteFilter, it fills the structure lazily.

I added shared equality deletes to Spark's RowDataReader and BatchDataReader by default, because the size of the shared structure is bounded by split size (multiplied by compression rate). To GenericReader I added a boolean parameter shareEqDeletes which is false by default, because the size is unbounded, and switching the parameter to true might cause OOMs. And I don't use Flink, so I hesitated to propose changing its behavior.

The non-shared DeleteFilter path is unchanged, so Flink and other engines are byte-for-byte identical. The shared path produces identical results (equivalence covered by tests).

Additionally, this PR changes task weight calculation algorithm - each delete file is accounted only once per bin even if it is referenced by several data files in that bin. This change can be moved into a separate PR if preferred.

In our case these changes reduced Spark memory consumption by 30-50% and, more important, prevented Spark from hanging for hours on building and evicting huge equality delete sets for each data file. For frequently updated tables (several thousands of equality delete files per day) ingested with Debezium Iceberg Sink rewrite_data_files execution time is reduced from several hours to several minutes.

Btw, I found that Trino uses a similar technique: EqualityDeleteFilter, DeleteManager.

[singhpk234]

wonder if you tried this already

#9563 ?

[kinolaev]

Hi @singhpk234 , I didn't try it, because I have only one big executor.

[github-actions]

This pull request has been marked as stale due to 30 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@iceberg.apache.org list. Thank you for your contributions.

[kinolaev]

The PR is not stale, it is awaiting review

[kinolaev]

The conflict is resolved

[kinolaev]

@huaxingao, could you please take a look at this PR when you get a chance? This is also related to equality deletes; it significantly improves read performance and memory consumption by lazily building a shared equality delete map for all data files in a Spark task.

[kinolaev]

Rebased, applied changes for Spark 4.2, and moved changes for other Spark versions to a separate branch for a backport PR after this one is merged.

[kinolaev]

@szehon-ho, could you please review this PR when you have a moment? I would also appreciate it if you could ask someone else interested in optimizing equality delete planning and reading for a review.

[kinolaev]

@aokolnychyi, could you please take a look at this PR when you get a chance? It improves equality delete read performance by building an equality delete structure shared across all data files in a task.