Repository navigation
feat(transaction): add UpdatePartitionSpecAction for partition spec evolution - #3441
Open
dhruvarya-db wants to merge 1 commit into
Open
dhruvarya-db wants to merge 1 commit into
dhruvarya-db wants to merge 1 commit into
Conversation
…volution Add `Transaction::update_partition_spec()` to add, remove and rename partition fields. The changes are resolved at commit time with the rules of Java's `BaseUpdatePartitionSpec`, so the same changes give the same spec as Java, including default field names and error messages. Field ids are assigned on the client. A new field that matches a field of an earlier spec reuses its id, preferring the highest id when several match; otherwise it takes the next id after `last-partition-id`. The commit sends `AddSpec` (plus `SetDefaultSpec` unless `add_non_default_spec` is used) and asserts the last assigned partition id and default spec id. Format version 1 tables are rejected for now. The spec builder still rejects specs that keep two time granularities of one column, so adding hour(ts) to a day(ts) table fails at commit on catalogs built on iceberg-rust. Co-authored-by: Isaac <no-reply@databricks.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Which issue does this PR close?
BaseUpdatePartitionSpeclogic to iceberg-rust #2583.What changes are included in this PR?
This adds
Transaction::update_partition_spec(). Until now the only way to evolve a partition spec was to build theAddSpecupdate by hand, choosing every field ID yourself.The logic is a port of Java's
BaseUpdatePartitionSpec, so the same changes give the same spec as Java, including the default field names (id_bucket_16,ts_day) and the error messages.Field IDs are assigned here rather than left to the catalog. A new field that matches one in an earlier spec reuses that field's ID, as the spec requires; a match means the same source column and transform, plus the same name if one is given. When several fields match, the highest ID wins, as in Java's
TableMetadata. Otherwise the field gets the next ID afterlast-partition-id. The commit sendsAddSpec, plusSetDefaultSpecunlessadd_non_default_spec()is called. It also asserts the last assigned partition ID and the default spec ID it started from.Not covered yet:
voidplaceholders, which needs its own handling.hour(ts)to aday(ts)table fails at commit on catalogs built on iceberg-rust. fix(spec): accept partition specs produced by spec evolution #3431 fixes the builder.Are these changes tested?
Yes. The unit tests port Java's
TestUpdatePartitionSpec. The table-level tests portTestTableUpdatePartitionSpec, committing through the memory catalog.AI Disclosure
Written with Claude Opus 5.5 and reviewed by me.