Repository navigation
cast string to boolean: trim ISO control bytes to match Spark's UTF8String.trimAll #4959
Description
Describe the bug
Comet's parse_utf8_to_boolean (native/spark-expr/src/conversion_funcs/string.rs) uses Rust str::trim, which strips only Unicode char::is_whitespace code points. Spark's UTF8String.trimAll() (referenced by Cast for stringToBool) trims every byte where Character.isWhitespace(b) || Character.isISOControl(b) (UTF8String.java:201-203, UTF8String.java:994-1010).
The two sets differ on the ASCII control bytes 0x00-0x08, 0x0E-0x1F, and 0x7F: Spark trims them, Comet does not.
Repro (per @mbutrovich on #4914)
`\" \x01true\x01 \"` -> Spark casts to `true`, Comet casts to `null` (or ANSI-errors).
Scope
Same divergence likely exists in other trim-based cast paths in conversion_funcs/string.rs. UTF8String.trimAll parity should be assessed systematically.
Prior context
Reported in the review of #4914; that PR intentionally left the pre-existing behavior alone to keep it a pure perf change.
Activity
test: cover byte-length arms with non-ASCII inputs; note trimAll gap
kazuyukitanimura commentedon Jul 20, 2026
Let's mark cast(string to boolean) as incompatible (narrow down if possible)
docs: document known correctness issues in the compatibility guide (#…
andygrove commentedon Jul 30, 2026
I dug into the "Scope" note above and the divergence turns out to be considerably wider than boolean, so I've opened #5149 as an epic and I'd suggest closing this one in favour of it.
Short version of what the audit found:
CAST(string AS date)is correct, becausedate_parser's localis_whitespace_or_iso_controlhappens to compute exactly Spark'strimAllset.str::trimalso trims non-ASCII Unicode whitespace (U+0085,U+00A0,U+1680,U+2000-U+200A,U+2028,U+2029,U+202F,U+205F,U+3000), which Spark never trims for any cast target. SoCAST(' true' AS boolean)returnstruein Comet andNULLin Spark, and under ANSI Comet silently succeeds where Spark throws. That's a silent-wrong-result path, arguably worse than the NULL-instead-of-value direction described here.trimAllset (bytes0x00-0x20and0x7F), but float / double / decimal go through JavaString.trim, which is charsU+0000-U+0020and does not include0x7F. So a single shared trim helper applied uniformly would fix boolean and integrals while introducing a new divergence in float and decimal.u8::is_ascii_whitespaceexcludes0x0B(vertical tab) whereasstr::trimincludes it, so the integral and decimal paths (which usetrim_ascii/is_ascii_whitespace) fail on0x0Bas well.Verified empirically in both directions: Spark ground truth via a Java probe against the 3.5.5 jars exercising the exact primitives each cast target uses (
UTF8String.trimAll,UTF8String.toInt/toLong,Double.parseDouble,new BigDecimal(String.trim),DateTimeUtils.stringToDate/stringToTimestamp), and Comet via the same 25-codepoint matrix throughspark_castin legacy and ANSI mode. The Spark code paths are identical in 3.5.8 and 4.1.1. Full matrix is in #5149.On the "mark cast(string to boolean) as incompatible" suggestion: I'd lean against it as the resolution here, since it would leave the other six targets silently wrong while singling out the one that was noticed first. All of them are
Compatible()inCometCast.canCastFromStringtoday. #5149 breaks the work down per target, and the two trim helpers it proposes should make each one a small change.andygrove commentedon Jul 30, 2026
@kazuyukitanimura @mbutrovich I'll go ahead and close this issue because #5149 now supersedes it (see previous comment).
docs: document known correctness issues in the compatibility guide (a…