You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
map_from_arrays falls back to Spark under mapKeyDedupPolicy=LAST_WIN, unlike map_from_entries #5589
With no CodegenDispatchFallback mixin, that means the whole projection falls back to Spark for any session running that policy.
CometMapFromEntries sits directly below it in the same file and already carries the mixin (:211-213) for its own analogous restriction, so the two closely related functions behave differently for no good reason.
MapFromArrays.doGenCode builds the map through ArrayBasedMapBuilder, which is where the dedup policy is actually implemented, so the dispatcher gets LAST_WIN right by construction. MapType is supported by CometBatchKernelCodegen.isSupportedDataType recursively.
Fix: mix CodegenDispatchFallback into CometMapFromArrays.
Not in scope: the separate null-key divergence documented in getCompatibleNotes and tracked as #4680. That one is a Compatible path today, so it is a correctness question rather than a fallback question — though it is worth noting that routing through the dispatcher would fix it too, since Spark's builder raises "Cannot use null as map key" itself.
Activity
LinSimon-901101 commentedon Sep 10, 2026
take