[blaginin]

related to #20052

https://github.com/apache/datafusion/actions/runs/22638279915/job/65619265884

-5-10 minutes

[AdamGS]

FWIW, we've also found mimalloc making a huge difference on macos, which I assume is a very popular dev environment

[comphead]

Thanks @blaginin however I'm not sure about mimalloc for tests tbh, we do mimalloc for benchmarks, but tests asserts correctness and mem usage for default user environment which is glibc malloc or musl malloc if we speaking for Linux and providing results from another mem allocator as a source of truth, WDYT?

[blaginin]

hey @comphead! In this pr i change just sqlogictests, and they don't assert mem usage - just correctness. They're pretty high level so I'd be surprised if the allocator interferes in any way

[comphead]

hey @comphead! In this pr i change just sqlogictests, and they don't assert mem usage - just correctness. They're pretty high level so I'd be surprised if the allocator interferes in any way

IMO I would still expect tests to be in the same environment, otherwise it would be a debugging nightmare in case of false positives or false negatives 🤔

[Dandandan]

Thanks @blaginin however I'm not sure about mimalloc for tests tbh, we do mimalloc for benchmarks, but tests asserts correctness and mem usage for default user environment which is glibc malloc or musl malloc if we speaking for Linux and providing results from another mem allocator as a source of truth, WDYT?

I am not sure tests should depend on the allocator? If they would it might be possible to change the allocator just for those tests?

[Omega359]

Thanks @blaginin however I'm not sure about mimalloc for tests tbh, we do mimalloc for benchmarks, but tests asserts correctness and mem usage for default user environment which is glibc malloc or musl malloc if we speaking for Linux and providing results from another mem allocator as a source of truth, WDYT?

Do people actually use the default allocator for their df based apps? I've found in my app that MiMalloc gave the best overall behaviour and was significantly better than glibc.

[comphead]

Thanks @Omega359 and @Dandandan — the motivation mostly comes from DuckDB, which uses jemalloc by default.
In practice this means:

• tests run with jemalloc
• the shipped DuckDB distribution also uses jemalloc

Docs:
https://duckdb.org/docs/stable/guides/performance/environment#memory-allocator
https://duckdb.org/docs/stable/core_extensions/jemalloc

The advantage is that tests and the shipped binary run with the same allocator. If we think mimalloc works better (we also use it in Comet), it might make sense to make it the default allocator everywhere so tests and production match, also update documentation.

WDYT?

[AdamGS]

what's production for DataFusion? Rust only allows you to set the global allocator once, setting it in a crate is a faux pas, and specifically here will block the use of the dhat allocator elsewhere

[Dandandan]

I think for tests it should be OK to switch to a different allocator.
They shouldn't fail when a different allocator would be used (we also expect now tests do not fail on user machines which use different system allocators).

You could always write tests that switch to a different allocator if necessary.

[github-actions]

Thank you for your contribution. Unfortunately, this pull request is stale because it has been open 60 days with no activity. Please remove the stale label or comment or this will be closed in 7 days.