Memory limit test smj_with_mem_limit_1 is flaky #26354

Describe the bug

memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1, added in #23988 and first released in 55.1.0, fails intermittently on 32-vCPU Linux hosts. In the failures the measured RSS growth is about 200 MB, well above the 160 MB allowance (40_000_000 * 4). Across repeated runs the result is bimodal: it either fits under the allowance or lands near 200 MB.

The test leaves little headroom even where it passes. On an Apple Silicon Mac it measures 149.4–149.8 MB against the 152.6 MiB allowance in five out of five runs, a margin of about 2%.

To Reproduce

On a 32-vCPU x86_64 Linux host (we see it on Amazon Linux 2023, EC2 c7a.8xlarge):

git checkout 55.1.0
cargo test --profile ci -p datafusion --features extended_tests --test core_integration -- \
  memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1_runner

The failure is intermittent, so repeat the run several times.

Example failure output:

Memory before: 140.9 MB, Memory after: 340.9 MB
Query: SELECT sum(rr.v) FROM generate_series(0, 0) AS l(k) JOIN (SELECT i % 1 AS k, i AS v FROM generate_series(1, 5000000) AS r(i)) rr ON l.k = rr.k, Memory usage: 200.0 MB, Memory limit: 152.6 MB

thread 'memory_limit::memory_limit_validation::smj_mem_validation::smj_with_mem_limit_1' panicked at datafusion/core/tests/memory_limit/memory_limit_validation/utils.rs:286:5:
Memory usage exceeded the theoretical limit. Actual: 200.0 MB, Expected limit: 152.6 MB

Measured usage in three failing runs: 197.8 MB, 199.7 MB, 200.0 MB. Four other runs of the same suite passed.

Expected behavior

The test passes deterministically on supported CI hosts, or its allowance leaves enough headroom for allocator variance between platforms.

Additional context

• Spilling is not the problem. The assertion that SortMergeJoinExec spilled passes in the failing runs, and smj_with_mem_limit_2 and smj_no_mem_limit pass.
• The baseline is much higher on Linux. The baseline query peaks at about 141 MB of RSS on the Linux host, against about 62 MB on macOS. The jump of roughly 50 MB between passing and failing runs suggests glibc malloc retaining an extra arena heap, which is 64 MB on 64-bit. We have not verified this, for example with MALLOC_ARENA_MAX.
• Our measurement helper was modified. We patch measure_max_rss in utils.rs to refresh only the current PID instead of calling System::new_all() inside the monitor task, and to seed the peak with a reading taken before the query. In our environment the full refresh could take long enough that the short baseline query finished before any sample, leaving the baseline RSS at 0 and turning the "delta" into absolute RSS. This may deserve a separate issue, and we can open a PR if that's useful.

Possible fixes:

• raise the allowance for smj_with_mem_limit_1, for example to 40_000_000 * 5;
• base the allowance on the spill metrics rather than a fixed RSS factor;
• run the memory-validation subprocesses with MALLOC_ARENA_MAX set, if the arena explanation holds.