Skip to content

feat(transaction): add UpdatePartitionSpecAction for partition spec evolution - #3441

Open
dhruvarya-db wants to merge 1 commit into
apache:mainfrom
dhruvarya-db:update-partition-spec-action
Open

dhruvarya-db wants to merge 1 commit into
apache:mainfrom
dhruvarya-db:update-partition-spec-action

Conversation

@dhruvarya-db

Copy link
Copy Markdown
Contributor

Which issue does this PR close?

What changes are included in this PR?

This adds Transaction::update_partition_spec(). Until now the only way to evolve a partition spec was to build the AddSpec update by hand, choosing every field ID yourself.

let tx = tx
    .update_partition_spec()
    .add_field("id", Transform::Bucket(16))
    .remove_field("category")
    .rename_field("ts_day", "day")
    .apply(tx)?;
let table = tx.commit(&catalog).await?;

The logic is a port of Java's BaseUpdatePartitionSpec, so the same changes give the same spec as Java, including the default field names (id_bucket_16, ts_day) and the error messages.

Field IDs are assigned here rather than left to the catalog. A new field that matches one in an earlier spec reuses that field's ID, as the spec requires; a match means the same source column and transform, plus the same name if one is given. When several fields match, the highest ID wins, as in Java's TableMetadata. Otherwise the field gets the next ID after last-partition-id. The commit sends AddSpec, plus SetDefaultSpec unless add_non_default_spec() is called. It also asserts the last assigned partition ID and the default spec ID it started from.

Not covered yet:

  • Format version 1 tables are rejected. They keep removed fields as void placeholders, which needs its own handling.
  • The spec builder rejects a spec that keeps two time granularities of one column. So adding hour(ts) to a day(ts) table fails at commit on catalogs built on iceberg-rust. fix(spec): accept partition specs produced by spec evolution #3431 fixes the builder.

Are these changes tested?

Yes. The unit tests port Java's TestUpdatePartitionSpec. The table-level tests port TestTableUpdatePartitionSpec, committing through the memory catalog.

AI Disclosure

Written with Claude Opus 5.5 and reviewed by me.

…volution

Add `Transaction::update_partition_spec()` to add, remove and rename
partition fields. The changes are resolved at commit time with the rules of
Java's `BaseUpdatePartitionSpec`, so the same changes give the same spec as
Java, including default field names and error messages.

Field ids are assigned on the client. A new field that matches a field of an
earlier spec reuses its id, preferring the highest id when several match;
otherwise it takes the next id after `last-partition-id`. The commit sends
`AddSpec` (plus `SetDefaultSpec` unless `add_non_default_spec` is used) and
asserts the last assigned partition id and default spec id.

Format version 1 tables are rejected for now. The spec builder still rejects
specs that keep two time granularities of one column, so adding hour(ts) to a
day(ts) table fails at commit on catalogs built on iceberg-rust.

Co-authored-by: Isaac <no-reply@databricks.com>

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add BaseUpdatePartitionSpec logic to iceberg-rust

2 participants