Datafusion 50 Performance Regression (array_has style filter/join for Parquet data set)

Describe the bug

I discovered a significant performance regression in the DataFusion 50 release. The essence of it is that filtering/joining based on array membership is no longer performant in the DF 50 series. This is a relative regression from DF 49, where this was reasonably performant.

NOTE: I do realize that this is not a particularly great design, and that we can't effectively filter on arrays (as far as I know?) like this using pushdown techniques or probably even row group metadata. Unfortunately I'm just a consumer of this data set, and this is the only way I found to do the job. Suggestions to improve the query are of course also welcome :)

To Reproduce

Here is SQL to reproduce in the datafusion-cli.

First, let's set up two external tables.

CREATE EXTERNAL TABLE categories_raw STORED AS PARQUET LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-09-09/categories/parquet/';

CREATE EXTERNAL TABLE places STORED AS PARQUET LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-09-09/places/parquet/';

Then, let's try doing a simple count of the places matching some subset of the categories (notes on the CTE in the additional context below).

Example query 1 (this is roughly our original query):

WITH categories_arr AS (
    SELECT array_agg(category_id) AS category_ids FROM categories_raw LIMIT 500
)
SELECT COUNT(*)
    FROM places p
    WHERE date_refreshed >= CURRENT_DATE - INTERVAL '365 days' AND array_has_any(p.fsq_category_ids, (SELECT category_ids FROM categories_arr));

Example query 2:

WITH allowed_categories AS (
    select * from categories_raw LIMIT 500
)
SELECT count(*)
FROM places p
JOIN allowed_categories ac ON array_has(p.fsq_category_ids, ac.category_id)
WHERE date_refreshed >= CURRENT_DATE - INTERVAL '365 days';

Expected behavior

TL;DR, I expect the queries to execute fairly quickly (low single digit number of minutes, with some reasonable variance for internet... I'm literally on the other side of the world from us-east-1 with a reasonably fast connection).

I also expect the query execution time, at least for this particular case, to not significantly vary based on the number of categories filtered.

Finally, I'd expect the CPU usage to either go down from current levels like DataFusion 49, OR if it's going to leverage more cores, to actually go faster ;) (DataFusion 49 rarely used more than a single core computing the count as described in Query 1, and did it much faster.)

Let me explain what I observed both queries doing under different DataFusion releases...

Query 1

Query 1 takes approximately 2 and a half minutes for me on DataFusion 49.0.2. This is the baseline and seems reasonable. Increasing the LIMIT to 1000 (more realistic), the execution time remains constant. CPU usage is moderate; it does not need to saturate all cores.

Under DataFusion 50, this just immediately consumes 100% of all CPU (not even bouncing; straight up redline haha), and doesn't seem to do much in the way of data transfer. I killed the query after around 20 minutes, since it was clear that it wasn't going to be able to complete in a reasonable time without any data transfer.

This behavior in DataFusion 50 occurs regardless of what the LIMIT value is. Even a LIMIT 10 sends all CPUs spinning, and no significant data transfer ever occurs.

I don't know how relevant this tidbit is, but unlike most flavors of high CPU utilization, this doesn't make my MacBook hot to the touch or even trigger fans. System load averages are around 100, but it's clearly not making any progress 🤷 No unusual memory activity either.

Query 2

I discovered Query 2 when looking for a workaround for DataFusion 50. It expresses a similar intent, and still uses an array operation (array_has in this case), but it's expressed more explicitly as a join condition rather than a weird subquery. However, it has wildly different performance characteristics.

On DataFusion 49, this consumes a fair bit of CPU, than Query 1, and it took around 22 minutes! That's like 10x longer than the original query.

DataFusion 50 fares a bit better better. It uses even more CPU than 49 does initially, and seems to start pulling from the network sooner/faster. It completes this query in just under 8 minutes.

So that's an improvement, right? Well... if you try doubling the LIMIT to 1000 (more representative of my typical query), the run time (DataFusion 50) approximately doubles to 15 minutes! This is a bit strange. But at least it finishes 😅

Reduced throughput streaming results

What I've included above is just the count which we do beforehand by calling .count() with a more useful select query. Once we start actually streaming the results, throughput is unsurprisingly slower than before. I'm seeing somewhere between 10% and (absolute best case 50%) of the former throughput (in records/second) through our pipeline.

Unfortunately the pipeline has too many downstream variables for me to cleanly disentangle that into an MRE without a lot of effort, but I guess I shouldn't be too surprised by this.

Additional context

Versions and test methods

I initially observed the regression on several crates.io releases as a dependency in our internal application (release build). I then reproduced using the respective datafusion-cli versions to make things easier to write up.

For DataFusion 49, I checked out the latest commit on branch-49, which is tagged as 49.0.2 and has hash f43df3f. I then built datafusion-cli in release mode.

For DataFusion 50, I used CLI version 50.2.0 via homebrew.

All observations are on an M1 Max MacBook Pro running macOS Sequoia 15.6.1.

I have not enabled any specific session context changes (code) or set any explicit options in the CLI.

Notes on category filter

The "real world" version of this query uses a computed VIEW with a long WHERE clause. A size-tunable simple CTE is simpler and better for the purpose of illustration though.

The "real" category filter includes about 1100 categories, and is a significant portion of the raw category list (1245 entries total). I have tested multiple variations of this, and there is (as one would expect) no difference between referencing the view and any random sample of 1100 categories.

Size of the array seems to matter?

array_has_any with a small cardinality list seems to be fine. For example, you can add (unresolved_flags IS NULL OR NOT array_has_any(unresolved_flags, ['inappropriate', 'duplicate'])) to the WHERE clause, it's fine. This actually may seem to contradict what I wrote above about LIMIT 10 not working on Query 1. Maybe there's something with static arrays vs computed ones? I have not had a chance to explore this yet.

I looked at the historyof array_has and I don't see any obvious culprits

https://github.com/apache/datafusion/commits/main/datafusion/functions-nested/src/array_has.rs

I am doing some profiling

@ianthetechie

I was able to reproduce this . Thank you for the report @ianthetechie

repo.sql

CREATE EXTERNAL TABLE categories_raw STORED AS PARQUET LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-09-09/categories/parquet/';

CREATE EXTERNAL TABLE places STORED AS PARQUET LOCATION 's3://fsq-os-places-us-east-1/release/dt=2025-09-09/places/parquet/';


WITH categories_arr AS (
    SELECT array_agg(category_id) AS category_ids FROM categories_raw LIMIT 500
)
SELECT COUNT(*)
    FROM places p
    WHERE date_refreshed >= CURRENT_DATE - INTERVAL '365 days' AND array_has_any(p.fsq_category_ids, (SELECT category_ids FROM categories_arr));

With 49.0.2:

andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ ~/Software/datafusion-cli/datafusion-cli-49.0.2 -f repro.sql
...
Elapsed 45.434 seconds.

With 50.2.0

andrewlamb@Andrews-MacBook-Pro-3:~/Downloads$ ~/Software/datafusion-cli/datafusion-cli-50.2.0 -f repro.sql
DataFusion CLI v50.2.0
...

I killed it after 3 minutes (didn't let it finish)

Some initial profiling suggests 50.0.2 is calling slice a lot somehow on an array:

I am looking

I thought this PR could be related,

feat: support FixedSizeList for array_has #16333

But both of these commits are slow (the are actually killed on my machine by the OOM killer)

I am git bisecting using this command

AWS_SKIP_SIGNATURE=true  cargo run --profile=release-nonlto --bin datafusion-cli -- -f ~/Downloads/repro.sql

Fascinating, it turns out to have nothing to do with array_has, it is related somehow to the NestedLoopsJoin

1206019 / Rewrite Nested Loop Join executor for 5× speed and 1% memory usage #16996 is the first bad commit

I will post a note on that ticket

1206019e1ad1fb923f85e3c62335de5ef75c683a is the first bad commit
commit 1206019e1ad1fb923f85e3c62335de5ef75c683a (HEAD)
Author: Yongting You <2010youy01@gmail.com>
Date:   Fri Aug 15 02:20:16 2025 +0800

    Rewrite Nested Loop Join executor for 5× speed and 1% memory usage (#16996)

    * Rewrite NestedLoopJoin for better performance and memory efficiency


    ---------

    Co-authored-by: Matt Butrovich <mbutrovich@users.noreply.github.com>

 benchmarks/README.md                                   |   78 +++--
 benchmarks/bench.sh                                    |   18 ++
 benchmarks/src/bin/dfbench.rs                          |    4 +-
 benchmarks/src/lib.rs                                  |    1 +
 benchmarks/src/nlj.rs                                  |  264 +++++++++++++++++
 datafusion/physical-plan/src/joins/nested_loop_join.rs | 2018 +++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++---------------------------------------------
 datafusion/physical-plan/src/joins/utils.rs            |   17 ++
 datafusion/sqllogictest/test_files/joins.slt           |    6 +-
 8 files changed, 1634 insertions(+), 772 deletions(-)
 create mode 100644 benchmarks/src/nlj.rs

Could you check the size of the inputs for the NLJoin? If this is NLJoin related it could be similar to #17547 and #17488 .

Could you check the size of the inputs for the NLJoin? If this is NLJoin related it could be similar to #17547 and #17488 .

I ran EXPLAIN ANAYZE

The relevant part of the plan I think is like this (tiny left input - 1 row, giant right input 22M rows)

NestedLoopJoinExec
   ProjectionExec (1 row)
   CoalesceBatchesExec (21917655 rows)

The full plan is here:

Full Output

+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| plan_type         | plan                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
| Plan with Metrics | ProjectionExec: expr=[count(Int64(1))@0 as count(*)], metrics=[output_rows=1, elapsed_compute=250ns]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|                   |   AggregateExec: mode=Final, gby=[], aggr=[count(Int64(1))], metrics=[output_rows=1, elapsed_compute=431.127µs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
|                   |     CoalescePartitionsExec, metrics=[output_rows=16, elapsed_compute=12.875µs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              |
|                   |       AggregateExec: mode=Partial, gby=[], aggr=[count(Int64(1))], metrics=[output_rows=16, elapsed_compute=20.151647ms]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                   |         ProjectionExec: expr=[], metrics=[output_rows=21018273, elapsed_compute=837.882µs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                   |           CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=21018273, elapsed_compute=17.333442058s]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                        |
|                   |             FilterExec: array_has_any(fsq_category_ids@0, category_ids@1), metrics=[output_rows=21018273, elapsed_compute=173.866690988s]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                   |               ProjectionExec: expr=[fsq_category_ids@1 as fsq_category_ids, category_ids@0 as category_ids], metrics=[output_rows=21917655, elapsed_compute=2.68371ms]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                      |
|                   |                 NestedLoopJoinExec: join_type=Right, metrics=[output_rows=21917655, elapsed_compute=35.600322924s, build_input_batches=1, build_input_rows=1, input_batches=2356, input_rows=21917655, output_batches=4696, build_mem_used=55068, build_time=7.882335ms, join_time=35.590911587s]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                           |
|                   |                   ProjectionExec: expr=[array_agg(categories_raw.category_id)@0 as category_ids], metrics=[output_rows=1, elapsed_compute=43.375µs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                         |
|                   |                     GlobalLimitExec: skip=0, fetch=500, metrics=[output_rows=1, elapsed_compute=42ns]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                       |
|                   |                       AggregateExec: mode=Single, gby=[], aggr=[array_agg(categories_raw.category_id)], metrics=[output_rows=1, elapsed_compute=79.125µs]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                   |
|                   |                         DataSourceExec: file_groups={1 group: [[release/dt=2025-09-09/categories/parquet/categories.zstd.parquet]]}, projection=[category_id], file_type=parquet, metrics=[output_rows=1245, elapsed_compute=1ns, batches_splitted=0, bytes_scanned=6295, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=2ns, metadata_load_time=89.590751ms, page_index_eval_time=2ns, row_pushdown_eval_time=2ns, statistics_eval_time=2ns, time_elapsed_opening=90.102958ms, time_elapsed_processing=7.179959ms, time_elapsed_scanning_total=51.312167ms, time_elapsed_scanning_until_data=51.186792ms]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                    |
|                   |                   CoalesceBatchesExec: target_batch_size=8192, metrics=[output_rows=21917655, elapsed_compute=80.199596ms]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                  |
|                   |                     FilterExec: CAST(date_refreshed@0 AS Date32) >= 2024-10-18, projection=[fsq_category_ids@1], metrics=[output_rows=21917655, elapsed_compute=1.918010644s]                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               |
|                   |                       DataSourceExec: file_groups={16 groups: [[release/dt=2025-09-09/places/parquet/places-00000.zstd.parquet:0..115950054, release/dt=2025-09-09/places/parquet/places-00001.zstd.parquet:0..119625610, release/dt=2025-09-09/places/parquet/places-00002.zstd.parquet:0..111979679, release/dt=2025-09-09/places/parquet/places-00003.zstd.parquet:0..99590199, release/dt=2025-09-09/places/parquet/places-00004.zstd.parquet:0..114422801, ...], [release/dt=2025-09-09/places/parquet/places-00006.zstd.parquet:15283909..123129325, release/dt=2025-09-09/places/parquet/places-00007.zstd.parquet:0..109503606, release/dt=2025-09-09/places/parquet/places-00008.zstd.parquet:0..116880554, release/dt=2025-09-09/places/parquet/places-00009.zstd.parquet:0..107480580, release/dt=2025-09-09/places/parquet/places-00010.zstd.parquet:0..112300670, ...], [release/dt=2025-09-09/places/parquet/places-00012.zstd.parquet:37168422..107444227, release/dt=2025-09-09/places/parquet/places-00013.zstd.parquet:0..110757380, release/dt=2025-09-09/places/parquet/places-00014.zstd.parquet:0..122112592, release/dt=2025-09-09/places/parquet/places-00015.zstd.parquet:0..119978222, release/dt=2025-09-09/places/parquet/places-00016.zstd.parquet:0..128256136, ...], [release/dt=2025-09-09/places/parquet/places-00018.zstd.parquet:44848422..120106873, release/dt=2025-09-09/places/parquet/places-00019.zstd.parquet:0..118004710, release/dt=2025-09-09/places/parquet/places-00020.zstd.parquet:0..118719840, release/dt=2025-09-09/places/parquet/places-00021.zstd.parquet:0..118914980, release/dt=2025-09-09/places/parquet/places-00022.zstd.parquet:0..120055444, ...], [release/dt=2025-09-09/places/parquet/places-00024.zstd.parquet:22110584..112529288, release/dt=2025-09-09/places/parquet/places-00025.zstd.parquet:0..117357063, release/dt=2025-09-09/places/parquet/places-00026.zstd.parquet:0..101323034, release/dt=2025-09-09/places/parquet/places-00027.zstd.parquet:0..113101190, release/dt=2025-09-09/places/parquet/places-00028.zstd.parquet:0..113015606, ...], ...]}, projection=[date_refreshed, fsq_category_ids], file_type=parquet, predicate=CAST(date_refreshed@0 AS Date32) >= 2024-10-18, metrics=[output_rows=105292526, elapsed_compute=16ns, batches_splitted=0, bytes_scanned=327606810, file_open_errors=0, file_scan_errors=0, files_ranges_pruned_statistics=0, num_predicate_creation_errors=0, page_index_rows_matched=0, page_index_rows_pruned=0, predicate_evaluation_errors=0, pushdown_rows_matched=0, pushdown_rows_pruned=0, row_groups_matched_bloom_filter=0, row_groups_matched_statistics=0, row_groups_pruned_bloom_filter=0, row_groups_pruned_statistics=0, bloom_filter_eval_time=230ns, metadata_load_time=41.780586406s, page_index_eval_time=15.576µs, row_pushdown_eval_time=230ns, statistics_eval_time=230ns, time_elapsed_opening=2.202947921s, time_elapsed_processing=3.475536283s, time_elapsed_scanning_total=695.001470448s, time_elapsed_scanning_until_data=64.328045286s] |
|                   |                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                             |
+-------------------+---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------+
1 row(s) fetched.
Elapsed 58.123 seconds.

Could you check the size of the inputs for the NLJoin? If this is NLJoin related it could be similar to #17547 and #17488 .

I ran EXPLAIN ANAYZE

The relevant part of the plan I think is like this (tiny left input - 1 row, giant right input 22M rows)
NestedLoopJoinExec
   ProjectionExec (1 row)
   CoalesceBatchesExec (21917655 rows)
The full plan is here:

Full Output

This 1 row in the left side is a lengthy aggregated array

The NLJ implementation is:

for each right_batch:
    for each left_row:
        join(left_row, right_batch)

and this join(left_row, right_batch) line will first repeat the left_row to the same length as right_batch, then apply the join filter. It's easier to be implemented this way due to the current filter API limitation.

This repeating/copying row step is done through to_array_of_size() in 50.0

datafusion/datafusion/physical-plan/src/joins/nested_loop_join.rs

Line 1668 in f199b00

scalar_value.to_array_of_size(filtered_probe_batch.num_rows())?

Note in 49.0 it's also copying the the row but with a different API. I suspect the slow down reason is: the 50.0 version is performing deep copy and 49.0 is doing shallow copy for this lengthy array in the left one-row input.

If that's the case, an easier fix would be making to_array_of_size() doing shallow copy on inner array/list. An alternative fix can be directly evaluating filter on row X batch, without repeating/copying the row to the same size as the batch.

#18161 is the direct fix to the issue, after that the performance should be similar to DF49

However, I think in some cases unnest can be used for the list column in the query, so that a much faster Hash Join operator can be applied, below is a query rewriting Q2 with unnest (I believe it needs an additional relatively cheap dedup step to make them equivalent), it's 300x faster on my machine

> WITH allowed_categories AS (
    select * from categories_raw LIMIT 500
),
unnested_places AS (
        select unnest(fsq_category_ids) as unnested_fsq_category_id
        from places
        WHERE date_refreshed >= CURRENT_DATE - INTERVAL '365 days'
)
SELECT count(*)
FROM unnested_places up
JOIN allowed_categories ac
on up.unnested_fsq_category_id = ac.category_id;

+----------+
| count(*) |
+----------+
| 7675969  |
+----------+
1 row(s) fetched.
Elapsed 0.435 seconds.

perf: Fix NLJ slow join with condition `array_has` (#18161)

perf: Fix NLJ slow join with condition `array_has` (apache#18161)

[branch-50] perf: Fix NLJ slow join with condition `array_has` (#18161)…

perf: Optimize array_has() for array needle

perf: Optimize array_has() for array needle (apache#23337)

Navigation Menu

Repository navigation

Datafusion 50 Performance Regression (array_has style filter/join for Parquet data set) #18070

Describe the bug

To Reproduce

Expected behavior

Query 1

Query 2

Reduced throughput streaming results

Additional context

Versions and test methods

Notes on category filter

Size of the array seems to matter?

7 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Datafusion 50 Performance Regression (array_has style filter/join for Parquet data set) #18070

Description

Describe the bug

To Reproduce

Expected behavior

Query 1

Query 2

Reduced throughput streaming results

Additional context

Versions and test methods

Notes on category filter

Size of the array seems to matter?

Activity

alamb commented on Oct 15, 2025

alamb commented on Oct 16, 2025

alamb commented on Oct 17, 2025

alamb commented on Oct 17, 2025

alamb commented on Oct 17, 2025

tschwarzinger commented on Oct 18, 2025

alamb commented on Oct 18, 2025

2010YOUY01 commented on Oct 19, 2025

2010YOUY01 commented on Oct 20, 2025

7 remaining items

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions