Repository navigation
Conversation
github-actions
Bot
added
arrow
ba2dcbd to
408a83f
Compare
|
Now that #10145 is fixed we can go back to this! |
408a83f to
34f46cd
Compare
34f46cd to
82fd9a1
Compare
|
Rebased on the latest main branch, there are too many changes, I need to have a self review, will make this ready when self review done. |
7a55774 to
d2e09e8
Compare
| use std::iter::zip; | ||
|
|
||
| #[test] | ||
| fn test_compatible_cast_logic_with_cast_kernel() { |
There was a problem hiding this comment.
Some cast logic was implemented directly here, add a test to cover it.
| ) { | ||
| Ok(Some(v)) => { | ||
| self.builder.append_value(v); | ||
| let $builder = &mut self.builder; |
There was a problem hiding this comment.
Add this because that variant_to_string return String now, but the builder.append_value receives a &str
9f05bca to
829e8cf
Compare
|
|
||
| fn write_timestamp( | ||
| /// Writes a timestamp value to the output using the given representation. | ||
| pub fn write_timestamp( |
There was a problem hiding this comment.
im not sure about adding this to our public api; we could probably get away with leaving it public and just copying the code required into the variant crate? especially as they seem to use default format + either no timezone or the default utc timezone
There was a problem hiding this comment.
realized there was a typo here; i meant get away with leaving it private*
There was a problem hiding this comment.
Changed; I meant to make the logic aligned with arrow-cast, even if we change the logic in arrow-cast later, but we have a test to cover this, and the logic is simple; copied to parquet-variant crate.
There was a problem hiding this comment.
Should we keep this private then?
There was a problem hiding this comment.
I think we need to update it to pub for now, previously, we'll call this function with different timezone/format now.
| /// let expected_d4 = NaiveDate::from_ymd_opt(2026, 6, 10).unwrap(); | ||
| /// assert_eq!(expected_d4, d4); | ||
| /// ``` | ||
| pub fn parse_date(string: &str) -> Option<NaiveDate> { |
There was a problem hiding this comment.
is this a leftover change? cant seem to find where this now being used in the diff
There was a problem hiding this comment.
This case just wants to show the supported format of this function; it is not used in the current change.
There was a problem hiding this comment.
so it doesnt need to be made public?
There was a problem hiding this comment.
Ah, this belongs to a future pr, leftover when rebasing and splitting the original pr. changed.
| // boolean -> string | ||
| let boolean_array = BooleanArray::from(vec![Some(true), Some(false)]); | ||
| let cast_array = cast(&boolean_array, &DataType::Utf8).unwrap(); | ||
| let boolean_utf8_array = cast_array.as_any().downcast_ref::<StringArray>().unwrap(); |
There was a problem hiding this comment.
| let boolean_utf8_array = cast_array.as_any().downcast_ref::<StringArray>().unwrap(); | |
| let boolean_utf8_array = cast_array.as_string::<i32>(); |
can use the downcasters here to make it more succinct: https://docs.rs/arrow/latest/arrow/array/trait.AsArray.html
|
@Jefffrey Thanks for the review, I've addressed the comments, please take another look when you're free. |
|
|
||
| fn write_timestamp( | ||
| /// Writes a timestamp value to the output using the given representation. | ||
| pub fn write_timestamp( |
There was a problem hiding this comment.
realized there was a typo here; i meant get away with leaving it private*
| /// let expected_d4 = NaiveDate::from_ymd_opt(2026, 6, 10).unwrap(); | ||
| /// assert_eq!(expected_d4, d4); | ||
| /// ``` | ||
| pub fn parse_date(string: &str) -> Option<NaiveDate> { |
There was a problem hiding this comment.
so it doesnt need to be made public?
| cast(×tamp_nano_arrow_array, &DataType::Utf8).unwrap(); | ||
| let timestamp_nano_cast_utf8_array = timestamp_nano_arrow_cast_array | ||
| .as_any() | ||
| .downcast_ref::<StringArray>() |
There was a problem hiding this comment.
fyi can replace more usages of this downcast pattern with just as_string::<i32>() in more of these tests, etc.
Jefffrey
left a comment
Jefffrey
left a comment
There was a problem hiding this comment.
should be good once ci is green
|
I'll take a look today |
| Variant::Int16(i) => Some(lexical_to_string(*i)), | ||
| Variant::Int32(i) => Some(lexical_to_string(*i)), | ||
| Variant::Int64(i) => Some(lexical_to_string(*i)), | ||
| Variant::Float(f) => Some(lexical_to_string(*f)), |
There was a problem hiding this comment.
arrow-cast's ArrayFormatter uses ryu for float
arrow-rs/arrow-cast/src/display.rs
Lines 712 to 724 in d5cd0da
the difference:
#[test]
fn reproduce_float_string_difference() {
let value = f32::from_bits(0xd378_62a7);
// What #10114 currently produces
let variant_output = lexical_to_string(value);
// What arrow-cast produces
let array = Float32Array::from(vec![value]);
let casted = cast(&array, &DataType::Utf8).unwrap();
let arrow_output = casted.as_string::<i32>().value(0);
assert_eq!(variant_output, "-1.066807e12");
assert_eq!(arrow_output, "-1066807000000.0");
// Fails:
assert_eq!(variant_output, arrow_output);
}| } | ||
|
|
||
| // convert a variant to an owned string. | ||
| pub(crate) fn variant_to_string(variant: &Variant<'_, '_>) -> Option<String> { |
There was a problem hiding this comment.
We should add FormatOptions parameter to pick how to write Date/Time/Timestamps to a string. It shouldn't be just default like it is in write_utc_timestamp_with_default_format and Date/Time arms.
| let _ = write!( | ||
| ret_str, | ||
| ", {}", | ||
| variant_to_string(&item).unwrap_or_default() |
There was a problem hiding this comment.
By calling variant_to_string recursively we drop the Variant::Object values inside a list.
arrow-cast doesn't support Objects -> String, but supports them inside a List via ArrayFormatter 🤷 (separate issue maybe?)
regardless of correctness of the Object support above, variant-cast to String shouldn't lose data.
#[test]
fn reproduce_list_of_objects_utf8_difference() {
// Build the Variant value: [{"x": 1}]
let mut variant_builder = VariantBuilder::new();
let mut variant_list = variant_builder.new_list();
variant_list
.new_object()
.with_field("x", 1_i32)
.finish();
variant_list.finish();
let (metadata, value) = variant_builder.finish();
let variant = Variant::new(&metadata, &value);
// Current #10114 result
let variant_output = variant_to_string(&variant).unwrap();
// Build the equivalent Arrow List<Struct<x: Int32>>
let fields = vec![Field::new("x", DataType::Int32, true)];
let struct_builder = StructBuilder::from_fields(fields, 1);
let mut arrow_list = ListBuilder::new(struct_builder);
let struct_builder = arrow_list.values();
struct_builder
.field_builder::<Int32Builder>(0)
.unwrap()
.append_value(1);
struct_builder.append(true);
arrow_list.append(true);
let arrow_list = arrow_list.finish();
// Arrow List<Struct> → Utf8 result
let casted = cast(&arrow_list, &DataType::Utf8).unwrap();
let arrow_output = casted.as_string::<i32>().value(0);
assert_eq!(variant_output, "[]");
assert_eq!(arrow_output, "[{x: 1}]");
// Fails
assert_eq!(variant_output, arrow_output);
}
|
@sdf-jkl Thank for the review, filed a new issue to track the Variant::Object in Variant::List case, I've update the code, and will run some benchmark to see if there are any performance regression in arrow-cast, will ping you for another review when ready. |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing variant-cast-utf8 (e73c88f) to aee9535 (merge-base) diff Run configurationrun benchmark cast_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
sdf-jkl
left a comment
sdf-jkl
left a comment
There was a problem hiding this comment.
Hey @klion26. The bench results are good on the cast kernel, but show regressions on the Variant one.
variant_get_utf8_from_date 1.00 570.2±5.55µs ? ?/sec 2.03 1159.8±8.32µs ? ?/sec
variant_get_utf8_from_decimal 1.00 476.4±5.35µs ? ?/sec 2.16 1029.4±5.10µs ? ?/sec
variant_get_utf8_from_float 1.00 449.0±2.24µs ? ?/sec 1.92 864.3±1.24µs ? ?/sec
variant_get_utf8_from_int 1.00 449.7±2.07µs ? ?/sec 1.65 741.3±1.33µs ? ?/sec
variant_get_utf8_from_list 1.00 657.8±15.06µs ? ?/sec 3.24 2.1±0.02ms ? ?/sec
variant_get_utf8_from_map_in_list 1.00 697.9±20.52µs ? ?/sec 5.75 4.0±0.01ms ? ?/sec
variant_get_utf8_from_time 1.00 780.9±2.99µs ? ?/sec 1.82 1424.1±7.84µs ? ?/sec
variant_get_utf8_from_timestamp_with_timezone 1.00 571.3±3.64µs ? ?/sec 2.56 1462.8±14.26µs ? ?/sec
variant_get_utf8_from_timestamp_without_timezone 1.00 572.0±4.01µs ? ?/sec 3.05 1744.1±4.25µs ? ?/sec
variant_get_utf8_from_unshredded_string 1.00 495.8±2.01µs ? ?/sec 1.42 705.0±4.03µs ? ?/sec
I think it's coming from the extra allocations that came from using String instead of &str
|
|
||
| fn write_timestamp( | ||
| /// Writes a timestamp value to the output using the given representation. | ||
| pub fn write_timestamp( |
There was a problem hiding this comment.
Should we keep this private then?
| if matches!(variant, Variant::Object(_)) { | ||
| return None; | ||
| } | ||
| let mut s = String::new(); |
| out.push_str(formats.null()); | ||
| true | ||
| } | ||
| Variant::String(s) => { | ||
| out.push_str(s); | ||
| true | ||
| } | ||
| Variant::ShortString(s) => { | ||
| out.push_str(s); | ||
| true | ||
| } | ||
| Variant::BooleanTrue => { | ||
| out.push_str("true"); | ||
| true | ||
| } | ||
| Variant::BooleanFalse => { | ||
| out.push_str("false"); |
| |capacity| -> B { B::with_capacity(capacity) }, | ||
| |value; temporal_formats: formats| variant_to_string(value, formats), | ||
| type_name: B::type_name(), | ||
| append_value: |builder, v| builder.append_value(&v) |
|
I'll take a look at the variant benchmark and get back |
e73c88f to
906d699
Compare
|
run benchmark variant_kernels |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing variant-cast-utf8 (906d699) to 891711e (merge-base) diff Run configurationrun benchmark variant_kernelsBENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing variant-cast-utf8 (906d699) to 891711e (merge-base) diff Run configurationrun benchmark variant_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
run benchmark variant_kernels |
|
🤖 Arrow criterion benchmark running (GKE) | trigger CPU Details (lscpu)Comparing variant-cast-utf8 (0db7289) to 891711e (merge-base) diff Run configurationrun benchmark variant_kernelsBENCH_COMMAND=cargo bench --features=arrow,async,test_common,experimental,object_store --bench variant_kernels File an issue against this benchmark runner |
|
🤖 Arrow criterion benchmark completed (GKE) | trigger Instance: Comparing variant-cast-utf8 (0db7289) to 891711e (merge-base) diff Run configurationrun benchmark variant_kernelsCPU Details (lscpu)Details
Resource Usagebase (merge-base)
branch
File an issue against this benchmark runner |
|
@sdf-jkl Please take another look when you're free, I think the perf is ok now ( |
| let value_str = d.integer().to_string(); | ||
| out.push_str(&format_decimal_str( | ||
| &value_str, | ||
| value_str.len(), | ||
| d.scale() as _, | ||
| )); | ||
| true |
There was a problem hiding this comment.
Could we use write_decimal here and in the Decimal8/Decimal16 arms? It was added in #11002 and is already available in this PR's base through arrow::datatypes. It formats the integer using a stack buffer and writes directly into out, avoiding the two temporary heap strings from integer().to_string() and format_decimal_str() for every decimal value.
| let value_str = d.integer().to_string(); | |
| out.push_str(&format_decimal_str( | |
| &value_str, | |
| value_str.len(), | |
| d.scale() as _, | |
| )); | |
| true | |
| arrow::datatypes::write_decimal(out, d.integer(), d.scale() as _).is_ok() |
This snippet fits the current bool return type. If we adopt the separate Result suggestion, propagate the write result with ? instead of .is_ok().
| pub(crate) enum VariantToStringArrowRowBuilder<'a, B: StringLikeArrayBuilder> { | ||
| Get(VariantToStringGetArrowBuilder<'a, B>), | ||
| Shred(VariantToStringShredArrowBuilder<'a, B>), | ||
| } |
There was a problem hiding this comment.
Could we follow the existing Boolean/numeric/timestamp/decimal builder pattern here: keep one string builder and one binary builder, each taking shred: bool, and drop the separate Get/Shred implementations and forwarding enums?
For binary, the shared builder can select value.as_u8_slice() when shredding and variant_to_binary(value) otherwise. For strings, both modes can share the existing String/ShortString fast path; other values should only go through string formatting when shred is false.
This preserves strict shredding behavior while removing the extra builder types, forwarding methods, and repeated constructor branches for the six string/binary array types.
| fn append_cast_value(&mut self, value: &Variant<'_, '_>) -> Result<bool> { | ||
| match variant_cast_with_options(value, self.cast_options, |value| { | ||
| variant_to_string(value, &self.formats) |
There was a problem hiding this comment.
A perf suggestion, can bench before and after.
We can have a reusable scratch: String on the string builder. variant_to_string currently creates a fresh String for each formatted value, which is then copied into the Arrow builder and dropped.
Calling scratch.clear() before each conversion, writing through write_variant_to_string, and appending scratch.as_str() on success would reuse the capacity across rows. The String/ShortString fast path can stay as-is. We would need to make the writer pub(crate) and preserve the wrapper's rejection of top-level Null/Object values (and the strict shredding guard if the builders are consolidated).
| Variant::Binary(v) => match std::str::from_utf8(v) { | ||
| Ok(s) => { | ||
| out.push_str(s); | ||
| true | ||
| } | ||
| Err(_) => false, |
There was a problem hiding this comment.
Could we use Arrow's hex formatting for binary values nested in collections? A list containing b"ab" and [0xff, 0x00] currently becomes [ab, ], whereas Arrow produces [6162, ff00]. Please add a comparison test for this case.
| if let Some(item) = iter.next() | ||
| && !write_variant_to_string(&item, formats, out) | ||
| { | ||
| out.push_str(formats.null()); |
There was a problem hiding this comment.
Could we return Result from the recursive formatter and use ?, rather than reducing formatting errors to bool and replacing failed children with null text? A date inside a list with %Y-%m-%d %H currently succeeds even with safe: false; the same format on a top-level date returns NULL with safe: true, while Arrow errors in both cases. Keeping formatting failures distinct from unsupported casts would fix both cases and remove the repeated true/.is_ok() branches. Please cover both cases with regression tests.
| pub(crate) fn variant_to_string<'v>( | ||
| variant: &Variant<'_, 'v>, | ||
| formats: &TemporalFormats<'_>, | ||
| ) -> Option<Cow<'v, str>> { |
There was a problem hiding this comment.
If we adopt the scratch-buffer suggestion, could we remove this allocating wrapper and its Cow machinery? The builder can use write_variant_to_string directly, retaining the string fast path and top-level Null/Object checks. The comparison tests could exercise the public conversion path instead.
For top-level Binary in Get mode, std::str::from_utf8(bytes) can also supply a borrowed &str directly to the builder, avoiding the extra copy through scratch. Invalid UTF-8 should retain safe/strict handling, shredding should still reject binary as a string, and nested binary should use Arrow's hex formatting.
| #[derive(Default)] | ||
| pub(crate) struct TemporalFormats<'a> { | ||
| date: CompiledTimeFormat<'a>, | ||
| time: CompiledTimeFormat<'a>, | ||
| timestamp: CompiledTimeFormat<'a>, | ||
| timestamp_tz: CompiledTimeFormat<'a>, | ||
| null: &'a str, | ||
| } |
There was a problem hiding this comment.
Optional cleanup: could we move TemporalFormats beside the formatting functions in type_conversion.rs? That would remove the reverse dependency from the conversion module to the builder module and let the formatter access these fields directly, without the five forwarding getters.
| #[test] | ||
| fn test_compatible_cast_logic_with_cast_kernel() { |
There was a problem hiding this comment.
Optional test cleanup: could we factor the repeated comparison setup into a helper that runs cast_to_variant -> variant_get and compares against Arrow casting? Reusing it for Utf8/LargeUtf8/Utf8View, nulls, and safe/strict failures would shorten these tests while covering the builder path as well as the formatter.
| ) -> FormatResult { | ||
| match &format.0 { | ||
| CompiledTimeFormatInner::Custom(items) => { | ||
| write!(f, "{}", self.format_with_items(items.0.iter()))? |
There was a problem hiding this comment.
Could we use self.format_with_items(items.0.iter()).write_to(f)? here, and the equivalent in the custom timestamp branches? Chrono's Display implementation allocates an intermediate String even when the destination buffer is reused; write_to writes directly to it. It can leave partial output on failure, so we should propagate the error and cover invalid formats, as discussed in the error-handling comment.
| (struct $name:ident<$lifetime:lifetime $(, $generic:ident: $bound:path )?> | ||
| |$array_param:ident $(, $field:ident: $field_type:ty)?| -> $builder_name:ident $(< $array_type:ty >)? { $init_expr: expr }, | ||
| |$value: ident $(, $shred: ident)?| $value_transform:expr, | ||
| |$value: ident $(, $shred: ident)? $(; temporal_formats: $formats:ident)?| $value_transform:expr, |
There was a problem hiding this comment.
Can we remove the temporal_formats macro parameter and its generated field, initialization, and local binding? No invocation uses it now that the string builder is handwritten and owns its formats directly.
|
Will take a look at the following days |
8584d79 to
a69f4b4
Compare
github-actions
Bot
added
the
arrow-kernels
# Conflicts: # Cargo.lock
a69f4b4 to
d0a1c59
Compare
Which issue does this PR close?
What changes are included in this PR?
arrow-rsAre these changes tested?
Are there any user-facing changes?