Repository navigation
Native implementation of get_json_object returns last value for duplicate keys, Spark returns first #4947
Description
Describe the bug
get_json_object diverges from Spark when a JSON object contains duplicate keys.
For input {"a":1,"a":2} and path $.a:
- Comet returns
2(the last occurrence) - Spark returns
1(the first occurrence)
Spark's GetJsonObjectEvaluator.evaluatePath (sql/catalyst/.../json/JsonExpressionEvalUtils.scala) stops at the first matching field once dirty is set, so it returns the first occurrence. Comet visits every entry and resolves to the last occurrence, matching serde_json's preserve_order overwrite semantics rather than Spark's.
This is a pre-existing divergence, not a regression. It predates the streaming rewrite in #4907 (the old serde_json + preserve_order path had the same last-wins behavior). It was noted during review of #4907.
Steps to reproduce
SELECT get_json_object('{"a":1,"a":2}', '$.a');
-- Spark: 1
-- Comet: 2Expected behavior
Match Spark and return the first occurrence of a duplicated key.
Additional context
The fix is on the Rust side in native/spark-expr/src/string_funcs/get_json_object.rs: SegmentVisitor::visit_map should stop updating found once a key has matched (while still consuming the remaining entries to validate the document). The test_duplicate_key_last_wins test added in #4907 would need to be flipped to first-wins.
Activity
[-]get_json_object returns last value for duplicate keys, Spark returns first[/-][+]Native implementation of `get_json_object` returns last value for duplicate keys, Spark returns first[/+]u70b3 commentedon Jul 17, 2026
take
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…
kazuyukitanimura commentedon Jul 20, 2026
Already marked as incompatible, downgrading the priority
fix: `get_json_object` returns first value for duplicate keys to matc…
6 remaining items
fix: `get_json_object` returns first value for duplicate keys to matc…
docs: document known correctness issues in the compatibility guide (a…
docs: drop compatibility notes for bugs that are now fixed (apache#5154)
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…
fix: `get_json_object` returns first value for duplicate keys to matc…