Skip to content

[EPIC] Improve NUMA-awareness #20902

Open
Feature
Open
[EPIC] Improve NUMA-awareness#20902
Feature
Labels
EPICA larger project, actively underway, with sub tasksperformanceMake DataFusion faster

Description

@Dandandan
Dandandan
opened on Mar 12, 2026

Is your feature request related to a problem or challenge?

This is a list to check where DataFusion w.r.t. NUMA-awareness and where we can improve.

I will list to some other tickets.

  • Create a pipeline/morsel scheduler #21719
  • Support morsel-splitting in Parquet, other datasources for better parallelisation / distribution / prefetching
  • Read parquet metadata to use ParquetMetaDataPushDecoder
  • Support/use pinning threads to cores to minimize context switches / cache misses)
  • Support reading into thread-local storage using async io / io_uring instead of using spawn_blocking
  • Split/decode row groups into sizes that fit well in cache (e.g. ~RecordBatch)
  • Better query execution pipelining / scheduling to minimize context switching / optimize data locality

Describe the solution you'd like

No response

Describe alternatives you've considered

No response

Additional context

No response

Activity

  1. Dandandan
    added
    performanceMake DataFusion faster
    EPICA larger project, actively underway, with sub tasks
    on Mar 12, 2026
  2. Dandandan
    added the
    Feature
    issue type on Mar 12, 2026

  3. nathanb9 commented on Mar 13, 2026

    @nathanb9
    nathanb9
    on Mar 13, 2026
    Contributor

    Hey @Dandandan looking for any help with any of these?

    I wanted to see impact of using ParquetMetaDataPushDecoder but looks like you're already working on that

  4. Dandandan commented on Mar 14, 2026

    @Dandandan
    Dandandan
    on Mar 14, 2026
    ContributorAuthor

    @nathanb9 Currently this is just me running some experiments and seeing what helps or gives non measurable improvement

    So far I tried the following (besides morsels):

    • Pinning threads - currently can't see a big diff in benchmarks. Could perhaps change with the morsel approach (as it has higher cpu usage), but perhaps OS scheduling does do the scheduling better or bandwidth has improved much since (and decompression / decoding perf hasn't) that it doesn't have any effect.
    • Blocking decode (i.e. local to thread) vs spawn_blocking the same, only minor changes.
    • Some prefetching IO: seems to help sometimes
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    EPICA larger project, actively underway, with sub tasksperformanceMake DataFusion faster

    Type

    Feature

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions