Repository navigation
Add limit to DefaultFileStatisticsCache #19052
Description
Also listing file statistics cache seems to not have any memory limit unlike metadata cache for example.
Is that by design , do you think we need to add similar limit for this cache too ?
Originally posted by @bharath-techie in #18971 (comment)
Basically the cache used in ListingTable comes from here:
https://github.com/apache/datafusion/blob/81512da2b0aaa474f6c4ba205b05eea7b3095176/datafusion/core/src/datasource/listing_table_factory.rs#L188-L187
Which somewhat unobviously sets a DefaultFileStatisticsCache here
https://github.com/apache/datafusion/blob/9f725d9c7064813cda0de0f87d115354b68d76e6/datafusion/catalog-listing/src/table.rs#L260-L259
The DefaultFileStatisticsCache has no limit:
https://github.com/apache/datafusion/blob/7d8b8602ad1be2f61f6a8ebb253ace9d85304ea7/datafusion/execution/src/cache/cache_unit.rs#L41-L40
Activity
alamb commentedon Dec 2, 2025
My suggestion would be to implement @nuno-faria 's ideas here: #18953 (comment)
And then once we had a real trait, add the ability to track memory used by Statistics and then use that measurement to implement a size limited cache
This would be similar in design to @BlakeOrth 's PR here
I suggest a sequence of smaller PRs:
Statistics::heap_sizethat reports the heap (allocation) size for statistics -- following the semantics of https://github.com/apache/arrow-rs/blob/b93fa52e47a83dff130709a6e80a4be5017dbf09/parquet/src/file/metadata/memory.rs#L35DefaultFileStatisticsCacheto have a memory limit and use the heap_size from 1 to constrain the sizedatafusion.runtime.statistics_cache_limit, similar todatafusion.runtime.metadata_cache_limit, and set the default statistics cache indatafusion/datafusion/execution/src/cache/cache_manager.rs
Lines 140 to 141 in 71fcd03
DefaultFilesMetadataCache)alamb commentedon Dec 2, 2025
It turns out @alchemist51 also pointed this out in
I didn't understand at first what was meant
jizezhang commentedon Dec 9, 2025
take
mkleen commentedon Dec 9, 2025
Dear @jizezhang I am already working on #19217, and this ticket is basically a requirement for it. I was wondering if it would be ok with you if I take it over. Let me know what you think.
mkleen commentedon Dec 9, 2025
However, I guess it's ok, we can merge it all together.
jizezhang commentedon Dec 9, 2025
Hey @mkleen feel free to take it if that makes things easier.
jizezhang commentedon Dec 10, 2025
@mkleen Could you try
takethis issue if you still plan to work on it? It looks like I cannot unassign myself...mkleen commentedon Dec 10, 2025
take
mkleen commentedon Dec 10, 2025
Thank you @jizezhang
alamb commentedon Dec 31, 2025
@mkleen and @jizezhang is there any update on this issue?
mkleen commentedon Dec 31, 2025
Yes, I am working on it as part of #19217. We would like to have this for the next release, right?
alamb commentedon Jan 1, 2026
Yes, please, let's try and get it done for 52.0.0 if possible (we are starting to test next week I believe)
If you can break the work into a few smaller PRs I will be more likely to be able to help review it quickly
abhita commentedon Jan 27, 2026
@mkleen @alamb
As the plan here is to enforce limit on
DefaultFileStatisticsCache, are we also planning on any mechanisms to handle stale entries here? perhaps through eviction or TTL?mkleen commentedon Jan 28, 2026
@abhita Currently it's handled by eviction see #20047 for details.