Skip to content

partition_to_path panics for identity/truncate fields whose source column was dropped #3381

Open

Description

@JeonDaehong

Apache Iceberg Rust version

v0.10.0

Describe the bug

Verified on main at 49b91d1 (the version field only lists releases).

PartitionSpec::partition_to_path panics when a partition field's source column is no longer in the schema and the field is identity or truncate over a non-string type.

#2773 made partition_type fall back to a source-independent result type for dropped source columns (Unknown for identity/truncate), so the partition_type(..).unwrap() succeeds. The panic happens later: to_human_string falls back to Display for Datum, which has no arm for an Unknown-typed int/long and hits unreachable!() (datum.rs).

Java's PartitionSpec.partitionToPath handles the same case: resultType uses UnknownType for a dropped source and toHumanString falls back to value.toString().

Old spec field, source dropped Rust Java
identity(int) panic p=1
identity(timestamp) panic p=1
truncate(int) panic p=10
identity(string), bucket, truncate(string), day, void ok ok

Today this is reachable through the public partition_to_path. #3253 also starts summarizing deleted files under their historical spec, which would put it on the commit path (see the discussion there).

To Reproduce

use iceberg::spec::{Literal, NestedField, PartitionSpec, PrimitiveType, Schema, Struct, Transform, Type};

let old_schema = Schema::builder()
    .with_fields(vec![
        NestedField::required(1, "x", Type::Primitive(PrimitiveType::Int)).into(),
        NestedField::required(2, "y", Type::Primitive(PrimitiveType::Int)).into(),
    ])
    .build()
    .unwrap();
let spec = PartitionSpec::builder(old_schema)
    .add_partition_field("y", "p", Transform::Identity)
    .unwrap()
    .build()
    .unwrap();

// `y` has been dropped from the current schema.
let current = Schema::builder()
    .with_fields(vec![NestedField::required(1, "x", Type::Primitive(PrimitiveType::Int)).into()])
    .build()
    .unwrap();

// Panics: internal error: entered unreachable code (datum.rs)
spec.partition_to_path(&Struct::from_iter([Some(Literal::int(1))]), current.into());

Expected behavior

No panic. Format the value as Java does for an unknown source type (p=1, p=10).

Willingness to contribute

I can contribute a fix for this bug independently

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    bugSomething isn't working

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions