Skip to content

[DISCUSSION] Sorts being removed from subqueries #15886

Closed
#21614
Closed
[DISCUSSION] Sorts being removed from subqueries#15886
#21614
Labels
discussionProject Discussion

Description

@maxburke
maxburke
opened on Apr 28, 2025

Describe the bug

Referenced discussion: https://the-asf.slack.com/archives/C01QUFS30TD/p1745875862723149

Given this table:

> create table d1 (ul_node_id string);

It appears that sorts are being removed from inner expressions. For example, here is a query with a sort expression:

> explain SELECT "ul_node_id", encode(ul_node_id, 'base64') FROM d1   ORDER BY encode(ul_node_id, 'base64') ASC NULLS LAST;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │          SortExec         │ |
|               | │    --------------------   │ |
|               | │ encode(d1.ul_node_id,Utf8(│ |
|               | │   "base64"))@1 ASC NULLS  │ |
|               | │            LAST           │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │ encode(d1.ul_node_id,Utf8(│ |
|               | │        "base64")):        │ |
|               | │ encode(ul_node_id, base64)│ |
|               | │                           │ |
|               | │        ul_node_id:        │ |
|               | │         ul_node_id        │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │          bytes: 0         │ |
|               | │       format: memory      │ |
|               | │          rows: 0          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.

If I wrap it in a no-op CTE the SortExec stage disappears:

> explain WITH t0 AS (SELECT "ul_node_id", encode(ul_node_id, 'base64') FROM d1   ORDER BY encode(ul_node_id, 'base64') ASC NULLS LAST) SELECT  * FROM t0;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │ encode(d1.ul_node_id,Utf8(│ |
|               | │        "base64")):        │ |
|               | │ encode(ul_node_id, base64)│ |
|               | │                           │ |
|               | │        ul_node_id:        │ |
|               | │         ul_node_id        │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │          bytes: 0         │ |
|               | │       format: memory      │ |
|               | │          rows: 0          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.

Manually hoisting out the ORDER BY from the CTE causes it to appear in the plan:

> explain WITH t0 AS (SELECT "ul_node_id", encode(ul_node_id, 'base64') FROM d1) SELECT * from t0   ORDER BY encode(ul_node_id, 'base64') ASC NULLS LAST;
+---------------+-------------------------------+
| plan_type     | plan                          |
+---------------+-------------------------------+
| physical_plan | ┌───────────────────────────┐ |
|               | │          SortExec         │ |
|               | │    --------------------   │ |
|               | │    encode(ul_node_id@0,   │ |
|               | │      base64) ASC NULLS    │ |
|               | │            LAST           │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       ProjectionExec      │ |
|               | │    --------------------   │ |
|               | │ encode(d1.ul_node_id,Utf8(│ |
|               | │        "base64")):        │ |
|               | │ encode(ul_node_id, base64)│ |
|               | │                           │ |
|               | │        ul_node_id:        │ |
|               | │         ul_node_id        │ |
|               | └─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐ |
|               | │       DataSourceExec      │ |
|               | │    --------------------   │ |
|               | │          bytes: 0         │ |
|               | │       format: memory      │ |
|               | │          rows: 0          │ |
|               | └───────────────────────────┘ |
|               |                               |
+---------------+-------------------------------+
1 row(s) fetched.

To Reproduce

No response

Expected behavior

No response

Additional context

No response

Activity

  1. maxburke
    added
    bugSomething isn't working
    on Apr 28, 2025

  2. comphead commented on Apr 28, 2025

    @comphead
    comphead
    on Apr 28, 2025
    Contributor

    Simpler test case

    ORDER BY in outer query

    > explain select x.* from (select 1 a union all select null) x order by a nulls last;
+---------------+------------------------------------------------------------+
| plan_type     | plan                                                       |
+---------------+------------------------------------------------------------+
| physical_plan | ┌───────────────────────────┐                              |
|               | │  SortPreservingMergeExec  │                              |
|               | │    --------------------   │                              |
|               | │      a ASC NULLS LAST     │                              |
|               | └─────────────┬─────────────┘                              |
|               | ┌─────────────┴─────────────┐                              |
|               | │         UnionExec         ├──────────────┐               |
|               | └─────────────┬─────────────┘              │               |
|               | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
|               | │       ProjectionExec      ││       ProjectionExec      │ |
|               | │    --------------------   ││    --------------------   │ |
|               | │            a: 1           ││          a: NULL          │ |
|               | └─────────────┬─────────────┘└─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
|               | │     PlaceholderRowExec    ││     PlaceholderRowExec    │ |
|               | └───────────────────────────┘└───────────────────────────┘ |
|               |                                                            |
+---------------+------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.009 seconds.

    should be the same as ORDER BY in CTE

    > explain select x.* from (select 1 a union all select null order by a nulls last) x;
+---------------+------------------------------------------------------------+
| plan_type     | plan                                                       |
+---------------+------------------------------------------------------------+
| physical_plan | ┌───────────────────────────┐                              |
|               | │         UnionExec         ├──────────────┐               |
|               | └─────────────┬─────────────┘              │               |
|               | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
|               | │       ProjectionExec      ││       ProjectionExec      │ |
|               | │    --------------------   ││    --------------------   │ |
|               | │            a: 1           ││          a: NULL          │ |
|               | └─────────────┬─────────────┘└─────────────┬─────────────┘ |
|               | ┌─────────────┴─────────────┐┌─────────────┴─────────────┐ |
|               | │     PlaceholderRowExec    ││     PlaceholderRowExec    │ |
|               | └───────────────────────────┘└───────────────────────────┘ |
|               |                                                            |
+---------------+------------------------------------------------------------+
1 row(s) fetched. 
Elapsed 0.008 seconds.

    But it is not. Sort is gone

  3. comphead
    added
    regressionSomething that used to work no longer does
    on Apr 28, 2025
  4. comphead
    changed the title [-]Sorts being removed from inner expressions[/-] [+][REGRESSION] Sorts being removed from inner expressions[/+] on Apr 28, 2025

  5. maxburke commented on Apr 28, 2025

    @maxburke
    maxburke
    on Apr 28, 2025
    ContributorAuthor

    I bisected this to commit 02eab80

  6. maxburke commented on Apr 28, 2025

    @maxburke
    maxburke
    on Apr 28, 2025
    ContributorAuthor

    I'm getting the results I expect if I revert the changes from that commit ^ in this file: datafusion/sql/src/relation/mod.rs (ie: remove the call to optimize_subquery_sort)

  7. maxburke
    added a commit that references this issue on Apr 28, 2025

    regarding apache#15886, comment out problematic query transformation

    5f3cb56

  8. comphead commented on Apr 28, 2025

    @comphead
    comphead
    on Apr 28, 2025
    Contributor

    I'm getting the results I expect if I revert the changes from that commit ^ in this file: datafusion/sql/src/relation/mod.rs (ie: remove the call to optimize_subquery_sort)

    Thanks for narrowing down

  9. ozankabak commented on Apr 29, 2025

    @ozankabak
    ozankabak
    on Apr 29, 2025
    Contributor

    I remember that we discussed this before and the conclusion that this is actually valid behavior. Quoting from Wikipedia:

    ORDER BY is the only way to sort the rows in the result set. Without this clause, the relational database system may return the rows in any order. If an ordering is required, the ORDER BY must be provided in the SELECT statement sent by the application. Although some database systems allow the specification of an ORDER BY clause in subqueries or view definitions, the presence there has no effect on the final result-set order, but makes sense when combined with a result offset clause or a fetch first clause. A view is a logical relational table, and the relational model mandates that a table is a set of rows, implying no sort order whatsoever.

    IIRC the SQL standard defines things way. Granted, a lot of people do not like this behavior and find it surprising, but it is not wrong AFAICT -- and a lot of DBs behave this way.

    Personally, I think the behavior is not surprising in @comphead's example, but it is somewhat surprising in @maxburke's example. For users that don't like the standard behavior, maybe we can add a configuration flag to DF that they can switch, which would make the optimize_subquery_sort function respect subqueries.

    I am removing the bug and regression tags for now -- we can reclassify as such if we find out I'm mistaken.

  10. ozankabak
    added
    discussionProject Discussion
    and removed
    bugSomething isn't working
    regressionSomething that used to work no longer does
    on Apr 29, 2025
  11. ozankabak
    changed the title [-][REGRESSION] Sorts being removed from inner expressions[/-] [+][DISCUSSION] Sorts being removed from inner expressions[/+] on Apr 29, 2025

  12. suibianwanwank commented on Apr 29, 2025

    @suibianwanwank
    suibianwanwank
    on Apr 29, 2025
    Contributor

    I agree that, to my knowledge, quite a few engines remove sorting in subqueries without LIMIT/OFFSET. Adding a configuration option is a good solution.

  13. alamb commented on Apr 29, 2025

    @alamb
    alamb
    on Apr 29, 2025
    Contributor

    I agree that, to my knowledge, quite a few engines remove sorting in subqueries without LIMIT/OFFSET. Adding a configuration option is a good solution.

    I also agree that the current behavior on main is expected but confusing. It would be great to improve it somehow (maybe a config setting)

  14. 4 remaining items

  15. nevi-me
    added a commit that references this issue on Jun 14, 2025

    regarding apache#15886, comment out problematic query transformation

    9caf823
  16. maxburke
    added 3 commits that reference this issue on Nov 26, 2025

    disable subquery sort optimization, see apache#15886

    0c500d4

    disable subquery sort optimization, see apache#15886

    c959970

    disable subquery sort optimization, see apache#15886

    c6ec94b

  17. adriangb commented on Dec 24, 2025

    @adriangb
    adriangb
    on Dec 24, 2025
    Contributor

    I hit this issue as well. I can confirm that Postgres, DuckDB, BigQuery, ClickHouse and SQLite do preserve the order from the subquery / CTE. I understand that technically this may be acceptable behavior, but I have not been able to find any system that has this behavior. @ozankabak @suibianwanwank could you share some of the database systems have this behavior?

    I would like to propose that we (1) make this configurable and (2) change the default since (a) the majority of popular systems seem to behave in the same way and (b) the safer behavior is to preserve the order by even if it has some performance cost.

  18. adriangb commented on Dec 24, 2025

    @adriangb
    adriangb
    on Dec 24, 2025
    Contributor

    I found that this causes what I would call a bug with Unparser: #19483.

    Essentially the following query unparses into a subquery which then when re-parsed and optimized looses the order by, i.e. parsing/unparsing SQL changes the query semantics:

    SELECT
  col * 2 as x_bucket,
  count(*)
FROM t1
GROUP BY x_bucket
ORDER BY x_bucket, count(*)

    We could try to fix it in the Unparser but the the point is that even our own codebase assumes the opposite behavior.

  19. ozankabak commented on Dec 25, 2025

    @ozankabak
    ozankabak
    on Dec 25, 2025
    Contributor

    Let's make this configurable. It is an optimization the other systems don't do. the users can then turn it on/off depending on their use case.

  20. github-merge-queue
    added a commit that references this issue on Dec 29, 2025

    Preserve ORDER BY in Unparser for projection -> order by pattern (#19483

    6ac7b89
  21. adriangb
    added a commit that references this issue on Jan 2, 2026

    Preserve ORDER BY in Unparser for projection -> order by pattern (apa…

    1ecae1f
  22. nevi-me
    added a commit that references this issue on Mar 23, 2026

    disable subquery sort optimization, see apache#15886

    e215b10
  23. github-merge-queue
    added 2 commits that reference this issue on Apr 16, 2026

    fix: impl `handle_child_pushdown_result` for `SortExec` (#21527)

    3b5008a

    feat: add a config to disable subquery_sort_elimination (#21614)

    7731130
  24. alamb
    closed this as completedin #21614on Apr 16, 2026
  25. Rich-T-kid
    added 2 commits that reference this issue on Apr 21, 2026

    fix: impl `handle_child_pushdown_result` for `SortExec` (apache#21527)

    0a45149

    feat: add a config to disable subquery_sort_elimination (apache#21614)

    97c2e0c
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    discussionProject Discussion

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions