Repository navigation
perf: Bloom, Count-Min, VarOpt, EBPPS and frequent items states in DuckDB's arena - #30
Merged
Merged
Conversation
Two overlay patches, so these sketches can keep all their memory in an arena allocator (DSAllocator's arena mode), which never frees: - count-min-hash-seeds-allocator.patch: count_min_sketch kept its hash seeds in a std::vector<uint64_t> with std::allocator, ignoring the sketch's allocator. The vector now uses the sketch's allocator (rebound to uint64_t). - frequent-items-purge-samples.patch: reverse_purge_hash_map::purge took its temporary median sample (up to 1024 values) from the sketch's allocator on every purge; it now lives on the stack. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…y can spill The Bloom filter and Count-Min aggregates (build, weighted build and merge, every input type including VARCHAR and BLOB) now keep their states in the aggregate arena, have no destructor, and export and import them as BLOBs, so they count toward memory_limit and DuckDB can spill them. A filter allocates its whole bit array and a Count-Min sketch all its counters up front, and DuckDB creates a state per input row when grouping reduces little, so a state first buffers up to 32 raw values (with their weights; strings are copied into the arena), bounded by the sketch's size, and builds the sketch only when more arrive. Both sketches are order-independent, so results are byte for byte the same. Buffered states export as their values, so importing them does not inflate them. The shared machinery (DSBufferedArenaState and its policies' interface, a weighted arena operation, scratch arenas for temporaries) is in src/include/datasketches_arena_buffer.hpp. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
The VarOpt and EBPPS aggregates over every item type but VARCHAR (whose std::string items own heap memory and keep the heap state) now keep their states in the aggregate arena and export and import them, so they can spill. States buffer their first 32 items with their weights (weights are checked as they arrive, with the library's messages); an EBPPS sketch reserves K items up front. A VarOpt state switches to a var_opt_union (in the arena) when it is first combined or merged, so later combines and merges update it in place instead of leaving a replaced sketch behind in the arena. The EBPPS merge aggregate reproduces the heap state's merge (the lighter sample is merged into the heavier, K the smaller of the two) without swapping a heap sketch into an arena state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… can spill A frequent items sketch keeps its items as std::string, which owns heap memory. The build and weighted build aggregates over numbers now keep their items as an inline key (the number's string, at most 24 bytes) that compares, hashes and serializes exactly as the std::string, in a sketch in the aggregate arena, so results are the same bytes and the states can spill. States buffer their first 32 items with their weights. VARCHAR items and the merge aggregates (whose input sketches may hold any strings) keep the heap state. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Runs the aggregates under a 128MB limit with 4 threads: many small (buffered) and larger groups, weighted variants, merges of stored sketches, EXPORT_STATE round trips and the VARCHAR heap paths. Bloom and Count-Min results are compared to the same query with plenty of memory, the others by exact facts. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Builds on #27.
frequent_items/_weightedvaropt/_weighted(k=64)ebpps/_weighted(k=64)bloom_filter(1000, 0.01)count_min/_weighted(3, 128)How the states work
src/include/datasketches_arena_buffer.hpp.std::string. VARCHAR inputs and the merge aggregates stay on the heap.Library patches (vcpkg overlay, port-version 4)
count-min-hash-seeds-allocator.patch:count_min_sketchkepthash_seedsin astd::vectorwithstd::allocator, ignoring the sketch's allocator.frequent-items-purge-samples.patch:reverse_purge_hash_map::purge()allocated a temporary array of up to 1024 values on every purge. The array is now on the stack.Known limits
std::vectorwith no allocator parameter.Tests
New
test/sql/datasketch_spill_items.test, under 128 MB with 4 threads. It covers many groups, the weighted variants, merges of stored sketches,EXPORT_STATEround trips and the VARCHAR heap paths. All tests pass.🤖 Generated with Claude Code