Repository navigation
[Variant] Preserve typed-value precedence without residual pre-rewriting #5980
Description
Remove the temporary residual masking in rewrite_shredding_state. Spark ignores value when a scalar/list typed_value is present; Arrow and Comet preparation currently decode those residual bytes unless Comet masks them first.
Required behavior
Spark's reconstruction, checked on 4.0.4, 4.1.3 and 4.2.0:
| State | Result |
|---|---|
Present typed scalar/list, malformed residual X'FF' |
Use typed value; never decode residual |
| NULL typed value, malformed residual | MALFORMED_VARIANT |
| Present typed object, malformed or non-object residual | MALFORMED_VARIANT |
| Residual field also declared in typed object schema | MALFORMED_VARIANT, even if the typed field is absent in this row |
Proposed change
Add opt-in NonObjectResidualPolicy::PreferTyped to the Arrow options API proposed in #5977. Preserve default rejection: the Parquet specification permits both representations only for partial objects.
Before residual parsing in handle_unshredded_case!, apply:
ignore_residual = PreferTyped && typed_value.is_valid(row) && !partial_object
When true, use the existing typed decoder. Invalid typed data still fails; it must not retry the residual. Apply this across primitive, decimal, timestamp and supported list builders. Keep object merging and schema-name conflict checks.
Give Comet's rewrite_residual_values the same row state so it skips validation, remapping and ordering of ignored residuals. If another row requires a rewritten buffer, copy ignored bytes unchanged. Keep root metadata validation and parent-null masking.
After the pinned Arrow dependency supports the option, enable it and remove the scalar/list is_not_null/nullif mask and its nullability adjustment. Both Arrow and Comet preparation must skip ignored bytes before the mask can be removed.
Completion
Test both policies with conflicting, Variant-null and malformed residuals; NULL/invalid typed values; partial objects and schema-name collisions; nested/sliced lists; mixed rewritten rows; and null parents. Compare native scan results/errors with supported Spark profiles, keeping upstream assertions unchanged.
Complete when the mask is deleted and these cases pass. Other preparation has separate removal conditions.
Parent: #5477. Related: #5977 (missing states), #5979 (metadata), #5978 (encoding). The options API is proposed.
Activity