Repository navigation
Establish whether the DataFusion 55 TPC-H off-heap bump is real, and changelog it if so #5704
Description
Describe the bug
The DataFusion 55 branch raises the off-heap size in CometTPCHQuerySuite from 2g to 4g:
conf.set(MEMORY_OFFHEAP_SIZE.key, "4g")It is not established whether DataFusion 55 genuinely needs more memory for these queries, or whether this is a consequence of the peak_mem_used accounting change in apache/datafusion#22710 that also disabled two CometAggregateSuite metric tests.
Expected behavior
If DataFusion 55 really does need more off-heap for TPC-H, that is a user-visible change and belongs in the changelog: anyone who sized their off-heap against TPC-H would be under-provisioned after upgrading. If it is only an accounting artifact, the bump should be reverted along with the metric fix.
Additional context
Found while reviewing #5262. Related to the ignored aggregate metric tests.
Activity
andygrove commentedon Sep 5, 2026
Tracked under #5709 ([EPIC] DataFusion 55 upgrade follow-ups).
test: restore the 2 GiB TPC-H memory budget (apache#5904)
andygrove commentedon Sep 15, 2026
Resolved by #5904. The full SF1
CometTPCHQuerySuitepasses on current main with the original 2 GiB off-heap budget, with both debug and release native libraries, so the increase to 4 GiB was not needed for this configuration and the suite is back at 2 GiB. With no user-visible change in memory requirements there is nothing to add to the changelog.One caveat from that PR: it did not measure the original DataFusion 55 upgrade commit or other workloads, so whether the bump was ever transiently necessary is unknown. That is historical at this point and not worth chasing separately. If a real memory regression on DataFusion 55 shows up elsewhere it should get its own issue with the workload attached.