Repository navigation
x > ALL (subquery) in the SELECT list fails with an internal schema error unless it is aliased #26622
Description
Describe the bug
A quantified comparison such as x > ALL (SELECT ...) or x = ANY (SELECT ...) fails to plan when it appears in the SELECT list without an alias. The same query works when the expression is aliased (AS r) and when it is used in WHERE. Using it inside an aggregate, for example bool_or(x > ALL (...)), fails with Schema error: No field named "bool_or(t.x > ALL)".
The error output itself says "This issue was likely caused by a bug in DataFusion's code."
To Reproduce
CREATE TABLE t(x INT) AS VALUES (1), (6);
CREATE TABLE s(y INT) AS VALUES (5);
SELECT x > ALL (SELECT y FROM s) FROM t;
-- duckdb 1.5.6: false, true
-- datafusion main: Internal error (below)
SELECT bool_or(x > ALL (SELECT y FROM s)) FROM t;
-- duckdb 1.5.6: true
-- datafusion main: Schema error: No field named "bool_or(t.x > ALL)"Actual output on main (datafusion-cli; schema dump shortened):
Error: Optimizer rule 'decorrelate_predicate_subquery' failed
caused by
Check optimizer-specific invariants after optimizer rule: decorrelate_predicate_subquery
caused by
Internal error: Assertion failed: compatible: Failed due to a difference in schemas: original schema: DFSchema { inner: Schema { fields: [Field { name: "t.x > ALL", data_type: Boolean, nullable: true }], ... }, new schema: DFSchema { inner: Schema { fields: [Field { name: "CASE WHEN EXISTS THEN Boolean(false) WHEN EXISTS THEN Boolean(NULL) ELSE Boolean(true) END", data_type: Boolean, nullable: true }], ... }.
This issue was likely caused by a bug in DataFusion's code. Please help us to resolve this by filing a bug report in our issue tracker: https://github.com/apache/datafusion/issues
Error: Optimizer rule 'optimize_projections' failed
caused by
Schema error: No field named "bool_or(t.x > ALL)".
Valid fields are "bool_or(CASE WHEN EXISTS THEN Boolean(false) WHEN EXISTS THEN Boolean(NULL) ELSE Boolean(true) END)".
For comparison, these work on the same build:
SELECT x > ALL (SELECT y FROM s) AS r FROM t; -- false, true
SELECT x FROM t WHERE x > ALL (SELECT y FROM s); -- 6
SELECT x IN (SELECT y FROM s) FROM t; -- false, falseExpected behavior
The first query should return false, true (as DuckDB 1.5.6 does), the same result as the aliased version. An optimizer rewrite should not change a plan's output schema.
Additional context
- Tested on
mainat c0ec512 (2026-10-22). Latest release 55.2.0 also fails: there, both queries fail inoptimize_projections. = ANYand< ANYfail the same way.- Cause:
RewriteSetComparison::rewrite_plan(datafusion/optimizer/src/rewrite_set_comparison.rs) replaces theSetComparisonwith aCASE WHEN EXISTS ...expression but does not keep the original expression name. Other rewriting rules useNamePreserverfor this. The output field of the Projection/Aggregate is renamed, and a later rule finds the old name missing. - Possibly related, but not the same bug: When performing set comparison subqueries (such as > ANY / > ALL / = ANY, etc.) on the same table (without using aliases), silently returns wrong results (0 rows or all rows). #25258 and
RewriteSetComparisonis overly aggressive rewriting some set comparison into multiple mark joins #25276 (other set-comparison issues), Physical plan does not support logical expression Exists #23022. - With the name preserved, the first query returns
false,true. Thebool_or(...)query then fails withPhysical plan does not support logical expression Exists. That is a separate limitation:SELECT bool_or(exists(SELECT y FROM s)) FROM tfails the same way onmain. - Found with an automated differential/metamorphic testing tool; reproduced on current
main.
Willingness to contribute
I would like to contribute a fix for this bug, but request guidance from the DataFusion community
Activity
yashaggarwal85d commentedon Oct 23, 2026
Hi @alpaylan, thanks for the detailed report and the pointer to
NamePreserver. I saw you mentioned wanting to fix this yourself. If you haven't started yet, I have a small fix ready that wraps the rewrite inRewriteSetComparison::rewrite_planwithNamePreserver, plus sqllogictest cases for> ALL,= ANY, and< ANYin the SELECT list. Would you mind if I open the PR? I'd leave thebool_or(...)variant (the "Physical plan does not support logical expression Exists" limitation) out of scope so it can be tracked separately, and I'm happy to step back if you already have something in progress.fix: preserve expression names in RewriteSetComparison
alpaylan commentedon Oct 23, 2026
Hi @yashaggarwal85d that's completely fine!
alamb commentedon Oct 28, 2026
I think we should implement support support for this feature as part of the more general subquery decorrelation project described here
in my opinion WE SHOULD NOT try and extend the existing subquery rewrites to support this type of query as that risks introducing many more wrong results, see #5492 (comment)