Skip to content

[EPIC] A collection of items to improve CASE performance #18075

Closed
Feature
Closed
[EPIC] A collection of items to improve CASE performance#18075
Feature
Labels
enhancementNew feature or requestperformanceMake DataFusion faster

Description

@alamb
alamb
opened on Oct 15, 2025

Is your feature request related to a problem or challenge?

@pepijnve and @chenkovsky has been working on a series of improvements to improve the performance of evaluating CASE expressions performance that I think should be highlighted / made more visible, so I am making a ticket to collect the work

TLDR

SELECT CASE WHEN ... ELSE ... END ...

Will go faster 🚀

Items

Activity

  1. alamb
    added
    enhancementNew feature or request
    on Oct 15, 2025
  2. alamb
    added the
    Feature
    issue type on Oct 15, 2025
  3. alamb
    added
    performanceMake DataFusion faster
    on Oct 15, 2025
  4. github-merge-queue
    added a commit that references this issue on Oct 20, 2025

    Short circuit complex case evaluation modes as soon as possible (#17898)

    7c215ed

  5. pepijnve commented on Oct 20, 2025

    @pepijnve
    pepijnve
    on Oct 20, 2025
    Contributor

    Copying this over from an arrow-rs discord discussion.

    Perhaps we can implement case when more efficiently using interleave. The current implementation strategy builds up a result one branch at a time as in this diagram (I made some mistakes in the labels, but I think the idea is clear enough).

    Diagram Image

    What if instead of merging after every branch we keep the partial results separate and instead update an index Vec<(usize,usize)> as we go. To be able to do that efficiently, we augment the record batch with a "row number" array that gets filtered along with the record batch itself. That gives us the offsets in the index vec to update. When evaluation is complete everything gets bundled up into the final result array with a single interleave operation.

    Diagram Image

  6. rluvaton commented on Oct 20, 2025

    @rluvaton
    rluvaton
    on Oct 20, 2025
    Member

    I improved zip in arrow for scalars by a lot:

    Currently waiting on review for the dependent PRs

  7. rluvaton commented on Oct 21, 2025

    @rluvaton
    rluvaton
    on Oct 21, 2025
    Member

    Added another massive case when improvement when the pattern is map like (when "aa" then 1 ...)

    PRs:

    it is 99% faster which means that you won't even notice that there is case when 😄

  8. pepijnve commented on Oct 23, 2025

    @pepijnve
    pepijnve
    on Oct 23, 2025
    Contributor

    @alamb could you add #18152 to the description list please?

  9. github-merge-queue
    added a commit that references this issue on Oct 28, 2025

    Optimize merging of partial case expression results (#18152)

    e9431fc

  10. pepijnve commented on Oct 28, 2025

    @pepijnve
    pepijnve
    on Oct 28, 2025
    Contributor

    @rluvaton if you're up for it, next one is ready to go #18329

  11. github-merge-queue
    added a commit that references this issue on Oct 31, 2025

    Project record batches to avoid filtering unused columns in `CASE` ev…

    c3e49fb
  12. tschwarzinger
    added 3 commits that reference this issue on Nov 2, 2025

    Short circuit complex case evaluation modes as soon as possible (apac…

    c455b4c

    Optimize merging of partial case expression results (apache#18152)

    f220445

    Project record batches to avoid filtering unused columns in `CASE` ev…

    c0efa37
  13. github-merge-queue
    added a commit that references this issue on Nov 5, 2025

    Avoid scatter operation in `ExpressionOrExpression` case evaluation m…

    32d2618
  14. codetyri0n
    added 3 commits that reference this issue on Nov 11, 2025

    Optimize merging of partial case expression results (apache#18152)

    cfeb8fa

    Project record batches to avoid filtering unused columns in `CASE` ev…

    3323552

    Avoid scatter operation in `ExpressionOrExpression` case evaluation m…

    def3939

  15. alamb commented on Nov 19, 2025

    @alamb
    alamb
    on Nov 19, 2025
    ContributorAuthor

    I am going to claim this particular EPIC is done for reporting purposes (though I realize we still gave #18183 outstanding)

  16. alamb
    closed this as completedon Nov 19, 2025
  17. EeshanBembi
    added a commit that references this issue on Nov 24, 2025

    Short circuit complex case evaluation modes as soon as possible (apac…

    f37bcb0
  18. github-merge-queue
    added a commit that references this issue on Dec 9, 2025

    perf: optimize CASE WHEN lookup table (2.5-22.5 times faster) (#18183)

    d85595e
  19. pepijnve
    added 3 commits that reference this issue on Feb 4, 2026

    Short circuit complex case evaluation modes as soon as possible (apac…

    ba6c06b

    Optimize merging of partial case expression results (apache#18152)

    250ec0c

    Project record batches to avoid filtering unused columns in `CASE` ev…

    33d16a3
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or requestperformanceMake DataFusion faster

    Type

    Feature

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions