Statistics update task drops all LRU caches when a small table goes from empty to a few rows and back; the next APPLY rebuilds structural caches #1972

Description

RecalculateAndUpdateStats (period 1 s) recalculates the statistics of a table after a "major" change. Once more than updateStatsDropLRUThreshold (default 1) major changes have accumulated, it calls ALRUMap.forceRemoveAllLRU(100). In 6.x the task is registered only in dev mode, so it affects every performance measurement on a dev-mode server (for example, one started from the IDE). In master the task has been registered in every mode since 07ea181 (2025-09-30, #1518), so in 7.0 the same will happen on production servers.

A materialized table that is normally empty is enough to trigger it, if creating a document adds rows to it and deleting the document removes them. Creating and then deleting one 20-line document gives two major changes (0 → 20, then 20 → 0), and all LRU caches of the server are dropped. The next APPLY of any document then rebuilds them.

Measured on 6.3 (build 6.3-20261104.141624-74), on one copy of an ERP database (50 GB), with the same sequence of steps through /eval. Dev mode was compared with -Dlsfusion.server.devmode=false; memory is the allocation of the APPLY thread.

What triggers the drop and what does not:

• Deleting a 1- or 4-line document is not a major change and drops nothing. Deleting a 20- or 100-line document does.
• In a numbered document the first stored write is the numerator, so the cost shows up on the numerator write. If the number is entered by hand, the next stored property written in that APPLY pays the same ~0.86 GB.

Reason

1. The counter counts recalculations, not the distance from the statistics the caches were built on. A table that goes 0 → 20 → 0 ends with the same estimate as before, but this still counts as two major changes and triggers the drop.
2. The drop also removes caches whose result does not depend on statistics. The ~0.8 GB node above is the rebuild of getOrderMapSingleApplyDepends and getDataChangeEvents. Both are @IdentityLazy, so they live in commonLruCache.
   • getDataChangeEvents is the set of data properties that have events. It is purely structural.
   • The dependency graph depends on statistics only through OldProperty.noUsePrevHeur → Property.hasAlotKeys, which compares the interface statistics with ALOT.reduce(2) (degree 5000). The statistics of real data never come close: 10^12 rows is degree ~17. So, as far as we can see, the graph is the same before and after any statistics update.
   • In the explain of that node, 385 MB of 816 MB are 584 calls of getInterfaceCostStat (the hasAlotKeys checks). The rest is the graph walk.
3. The drop is not logged, either by the task or by Service.updateStats. It shows up only as an unexplained slow APPLY, and lru.log logs only the memory tuner.

Suggested changes

1. Count a table or class as changed only when its estimate has moved away from its estimate at the last drop — by majorStatChangeDegree — rather than on every recalculation. Alternatively, ignore changes where both the old and the new row counts are below a small threshold, for example statDegree ^ majorStatChangeDegree = 25 rows. Either way, creating and deleting the same document no longer drops anything.
2. Keep the structural caches out of forceRemoveAllLRU: getDataChangeEvents, getOrderMapSingleApplyDepends, or at least the hasAlotKeys results. Alternatively, rebuild them only when a hasAlotKeys result can change. This removes ~0.8 GB and ~1 s from the first APPLY after every drop, including the drops made by Service.updateStats.
3. Log every drop by statistics (for example to lru.log), with the tables and classes and their old and new estimates.