Skip to content

fix(spec): read float JSON values like iceberg-java - #3476

Open
spicysharkco wants to merge 1 commit into
apache:mainfrom
spicysharkco:fix/float-json-single-value
Open

spicysharkco wants to merge 1 commit into
apache:mainfrom
spicysharkco:fix/float-json-single-value

Conversation

@spicysharkco

Copy link
Copy Markdown

Which issue does this PR close?

What changes are included in this PR?

RawLiteral::try_into for PrimitiveType::Float rejected a double when (f64::from(v as f32) - v).abs() > f32::EPSILON. That is an absolute bound, so valid floats such as 100.1 and 123.456 failed with Unable to convert raw literal (double) fail convert to type float for: type mismatch. This path reads float partition values in deserialize_data_file_from_json and float Datum values.

The check is replaced by a plain v as f32. That is what iceberg-java does: SingleValueParser.fromJson checks isFloatingPointNumber() for FLOAT and returns floatValue() (SingleValueParser.java L73-L79). It is also what Literal::try_from_json already does for field defaults. A double with no exact f32 now narrows to the nearest one: 16777217.0 gives 16777216.0, where main returned an error.

Doubles outside the f32 range (for example 1e39, which becomes infinity) behave as on main and as floatValue(). That is the subject of #3333 and is not touched here.

Notes for reviewers:

Are these changes tested?

Unit tests in crates/iceberg/src/spec/values/tests.rs, which fail on current main and pass with this change:

  • raw_literal_json_number_to_float: 100.1, 123.456, -123.456, f32::MAX, f32::MIN and the smallest subnormal, parsed with serde_json::from_str::<RawLiteral>, convert to exactly that f32.
  • raw_literal_avro_double_to_float: the same values through Value::Double, both as the widened f32 and as the shortest decimal parsed to f64.
  • raw_literal_double_without_exact_float_narrows_to_nearest: 16777217.0 becomes 16777216.0, and 1.0000000001 becomes 1.0.

The Float branch of RawLiteral::try_into rejected a double whose
distance to the narrowed f32 exceeded the absolute f32::EPSILON, so
valid values such as 100.1 and 123.456 failed. iceberg-java narrows
with floatValue() and does no precision check; do the same.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant