Repository navigation
Support loading and rewriting file-scoped position deletes #3287
Description
Is your feature request related to a problem or challenge?
Iceberg Rust has basic position-delete file writing support, but row-level rewrite workflows also need to:
- load positions from an existing file-scoped position-delete file
- validate that the delete file actually targets the expected data file
- merge existing and newly produced delete positions
- write the resulting positions back in Iceberg-required "(file_path, pos)" order
- de-duplicate repeated positions
One motivating use case is native Iceberg merge-on-read row-level writes in Apache DataFusion Comet, where an execution engine may need to read existing position deletes for a data file, combine them with newly generated deletes, and write a canonical replacement delete file.
The functionality is generic to Iceberg Rust and should also be useful for other row-level write and delete-file rewrite implementations.
Describe the solution you'd like
Position delete index loader
Add a loader for a single file-scoped V2 Parquet position-delete file that:
- resolves "file_path" and "pos" using the reserved Iceberg field IDs
- validates the physical schema and delete-file metadata
- verifies all physical rows target the expected data file
- validates non-negative positions and record counts
- canonicalizes duplicate/out-of-order positions into a sorted position index
This should stay separate from the existing scan-oriented delete loader, since rewrite callers require the stronger invariant that the entire supplied delete file belongs to one data file.
Sorting position-only delete writer
Add a writer for unordered "(file_path, pos)" input that:
- buffers positions by data-file path
- de-duplicates positions
- writes rows sorted by "file_path", then "pos"
- supports both "RecordBatch" input and individual position insertion
- preserves Iceberg position-delete metric semantics:
- reserved-column counts are omitted
- bounds are retained for file-scoped delete files
- bounds are removed when a physical delete file references multiple data files
The initial implementation can remain in-memory; spilling can be added separately if needed.
Scope
This is intended for V2 position-delete files. It does not add V3 deletion-vector writing or transaction-level row-delta commit handling.
The writer also does not need to own discovery of previous delete files. Callers can load existing file-scoped deletes using the position-delete index loader and feed those positions into the sorting writer together with newly generated deletes.
Willingness to contribute
I can contribute to this feature independently
Activity
unikdahal commentedon Sep 26, 2026
take