COALESCE expr in datafusion should perform lazy evaluation of the operands

Describe the bug

We observed that datafusion-spark's coalesce should perform lazy evaluation of the operands from left to right similar to spark . Current implementation throws an exception in case any operand throws an error which is not in alignment with spark which does lazy evaluation to return first non NULL values

To Reproduce

select coalesce(100, 1/0) from tbl

Expected behavior

result : 100 but throws an exception

Additional context

No response

@andygrove

cc : @andygrove , @kazuyukitanimura

For reference, this is the behavior in Postgres:

postgres=# select coalesce(1, 1/0);
 coalesce 
----------
        1
(1 row)

postgres=# select coalesce(1/0, 1);
ERROR:  division by zero

This differs from DataFusion's behavior:

DataFusion CLI v48.0.0
> select coalesce(1, 1/0);

Arrow error: Divide by zero error

@coderfender

@coderfender This issue is for DataFusion's coalesce expression and is not specific to datafusion-spark as far as I know.

Ah that explains it (fixed the title) . ~~Perhaps we are adhering to postgreSQL 's coalesce implementation ?~~

I can work on this once the comet's side impl is fixed .

We merged a fix for this in the 51 release, but we have found as subtle problem so I am planning to revert it for 51 (the fix should appear in 52):

[branch-51] Revert rewrite for coalesce, nvl and nvl2 simplification #18567

@comphead

Reviving this thread to update comet side of things once we bump DF version to 52 @comphead , @andygrove

This query works well now in datafusion 52

DataFusion CLI v52.1.0
> select coalesce(100, 1/0)  from values (1), (2);
+------------------------------------------+
| coalesce(Int64(100),Int64(1) / Int64(0)) |
+------------------------------------------+
| 100                                      |
| 100                                      |
+------------------------------------------+
2 row(s) fetched.
Elapsed 0.038 seconds.

@alamb

Thank you for the update @alamb . I will revert comet's fix to use DF's coalesce function then

Navigation Menu

Repository navigation

COALESCE expr in datafusion should perform lazy evaluation of the operands #17322

Describe the bug

To Reproduce

Expected behavior

Additional context

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

COALESCE expr in datafusion should perform lazy evaluation of the operands #17322

Description

Describe the bug

To Reproduce

Expected behavior

Additional context

Activity

coderfender commented on Aug 25, 2025

andygrove commented on Aug 25, 2025

andygrove commented on Aug 25, 2025

coderfender commented on Aug 25, 2025

coderfender commented on Aug 26, 2025

alamb commented on Nov 9, 2025

coderfender commented on Feb 8, 2026

alamb commented on Feb 9, 2026

coderfender commented on Mar 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions