No validation that prevents creating a V2 table with timestamp_ns/timestamptz_ns types #2418

Activity

1. mfellner commented on Oct 11, 2026

   

   mfellner

   on Oct 11, 2026

   More actions

   Related to #2411 (v3 tracking).

   Re-checked against main at a766783.

   Current schema: covered on main. Creating a v2 table, or loading v2 metadata whose current schema contains timestamp_ns, timestamptz_ns, variant or unknown, now fails with:

   DataInvalid => Invalid schema for v2:
   - Invalid type for x: timestamp_ns is not supported until v3 but format version is v2.
   

   The type floors and this check were added in #2188 (merged 2026-07-16). According to the git tags, #2188 is in v0.11.0-rc.1 and later, and not in v0.10.1. unknown was added later in #2773, which is not in any tag up to v0.11.0-rc.2. #2417, which referenced this issue, was closed without merging with the comment "Covered by #2188".

   Remaining gap: non-current schemas. The check runs only on the current schema. TableMetadata::try_normalize calls validate_schema_format_compatibility, which checks current_schema() only. As a result, on a v2 table:

   • TableMetadataBuilder::add_schema(..) followed by build() returns Ok when a non-current schema contains timestamp_ns, timestamptz_ns, variant or unknown.
   • Deserializing v2 metadata JSON that contains such a historical schema succeeds.

   Java rejects the first case. TableMetadata.Builder.addSchemaInternal calls Schema.checkCompatibility(schema, formatVersion), which throws IllegalStateException. This comes from reading apache/iceberg main, not from running Java. Java's metadata parser does not run this check on the schemas it reads, so the difference is limited to the builder path.

   Expected: add_schema rejects a schema that uses types the table's format version does not support, as Java does. Whether loading should also reject historical schemas is for maintainers to decide, since Java's parser accepts them.

   Maintainers could close this issue and track the non-current-schema gap separately, or retitle this one to cover it.

   Repro (scratch copy of a766783, integration test using only the public API)

   use iceberg::spec::{NestedField, PrimitiveType, Schema, TableMetadata, TableMetadataBuilder, Type};
   
   const V2_METADATA: &str = include_str!("../testdata/table_metadata/TableMetadataV2Valid.json");
   
   #[test]
   fn builder_add_schema_with_timestamp_ns() {
       let base: TableMetadata = serde_json::from_str(V2_METADATA).unwrap();
       let schema = Schema::builder()
           .with_schema_id(2)
           .with_fields(vec![
               NestedField::optional(2, "ts", Type::Primitive(PrimitiveType::TimestampNs)).into(),
           ])
           .build()
           .unwrap();
       let res = TableMetadataBuilder::new_from_metadata(base, None)
           .add_schema(schema)
           .and_then(|b| b.build());
       println!("BUILDER ok={} err={:?}", res.is_ok(), res.as_ref().err().map(|e| e.to_string()));
   }
   
   #[test]
   fn deserialize_v2_metadata_with_historical_timestamp_ns_schema() {
       let mut v: serde_json::Value = serde_json::from_str(V2_METADATA).unwrap();
       v["schemas"].as_array_mut().unwrap().push(serde_json::json!({
           "type": "struct",
           "schema-id": 2,
           "fields": [{"id": 2, "name": "ts", "required": false, "type": "timestamp_ns"}]
       }));
       let res = serde_json::from_value::<TableMetadata>(v);
       println!("DESER ok={} err={:?}", res.is_ok(), res.as_ref().err().map(|e| e.to_string()));
   }

   Output on main (cargo test --offline -p iceberg --test repro_2418_min -- --nocapture):

   BUILDER ok=true err=None
   DESER ok=true err=None
   

   The same two tests with timestamptz_ns, variant and unknown in place of timestamp_ns also return ok=true.

   Found during an agent-assisted audit of v3 support (see #2411). The reproduction above was run against main at a766783.