Repository navigation

Intermediate result blocked approach to aggregation memory management - #15591

Draft

Rachelint wants to merge 102 commits into

apache:mainfrom

Rachelint:intermeidate-result-blocked-approach

Draft

Intermediate result blocked approach to aggregation memory management#15591
Rachelint wants to merge 102 commits into
apache:mainfrom
Rachelint:intermeidate-result-blocked-approach

Commits on Apr 23, 2025
- define the needed methods in `GroupAccumulator` and `GroupValues`.
  Rachelint
  committed
  4353748
- define `GroupIndexOperations`, and impl block/flat mode for it.
  Rachelint
  committed
  a4450f7
- support block approach for `GroupValuesPrimitive`.
  Rachelint
  committed
  96b7435
- add new emit mode.
  Rachelint
  committed
  38d4fc6
- make `NullState` codes common for better reuse.
  Rachelint
  committed
  72e0fc3
- improve comments.
  Rachelint
  committed
  2dba944
- remove stale codes.
  Rachelint
  committed
  c08f23a
- add tests.
  Rachelint
  committed
  f6f3bd6
- support dynamic dispatching for `NullState`.
  Rachelint
  committed
  53e8c8e
- improve tests to cover `BlockedNullState`.
  Rachelint
  committed
  b5d231e
- complete the impl of blocked `GroupValuesPrimitive`.
  Rachelint
  committed
  a644b17
- support blocked mode for `GroupValuesPrimitive`.
  Rachelint
  committed
  67f03fc
- make `EmitTo::NextBlock` simpler.
  Rachelint
  committed
  d165fb0
- extract the common codes of block for reusing.
  Rachelint
  committed
  319e135
- support blocked mode for `PrimitiveGroupsAccumulator`.
  Rachelint
  committed
  808f142
- impl block based result returning logic.
  Rachelint
  committed
  489f093
- add judgement about when we should enable blocked groups optimization.
  Rachelint
  committed
  54002a1
- add config to control if we enable blocked groups optimization.
  Rachelint
  committed
  1f7b4bc
- fix e2e sql tests.
  Rachelint
  committed
  e58afa5
- fix group values len method.
  Rachelint
  committed
  4294ab7
- add `memory_limit` to expose the info in `MemoryPool`.
  Rachelint
  committed
  79714a4
- modify the spilling judgement.
  Rachelint
  committed
  266b48e
- add unit tests for primitive group values.
  Rachelint
  committed
  55de98c
- improve comments for `GroupIndexOperations`.
  Rachelint
  committed
  9145833
Commits on Apr 24, 2025
- add `enable_aggregation_blocked_groups` to aggr fuzzy test.
  Rachelint
  committed
  04f15b0
Commits on Apr 25, 2025
- refactor and make `QueryBuilder` more configurable.
  Rachelint
  committed
  be64a74
- fix tests.
  Rachelint
  committed
  7da0259
Commits on Apr 27, 2025
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  d771038
- fix clippy.
  Rachelint
  committed
  ffb11cd
- add fuzzy tests for blocked groups, and fix enable logic.
  Rachelint
  committed
  7f543d8
- update config.md and fix clippy.
  Rachelint
  committed
  868210f
- improve comment about blocked groups for `GroupedHashAggregateStream`.
  Rachelint
  committed
  bdcd1b8
- fix stack overflow.
  Rachelint
  committed
  3c7317d
- add extended query to see the improvement.
  Rachelint
  committed
  ff9c3ad
Commits on May 3, 2025
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  96b3c77
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  3e23408
- Update datafusion/common/src/config.rs
  
  Rachelint
  and
  alamb
  authored
  bb63628
- update config.
  Rachelint
  committed
  a7c4c7b
- fix fmt.
  Rachelint
  committed
  e033567
- fix logic test.
  Rachelint
  committed
  d173056
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  29222e1
- improve comments.
  Rachelint
  committed
  426e2ee
- move group index operations to a new module.
  Rachelint
  committed
  948c4ce
Commits on May 4, 2025
- define `Blocks` and use it to refactor.
  Rachelint
  committed
  09b97ab
- extract blocks to a dedicated module.
  Rachelint
  committed
  cee016c
- add tests for `Blocks`.
  Rachelint
  committed
  75ee3f3
Commits on May 5, 2025
- simplify codes about blocks.
  Rachelint
  committed
  5a6e030
- fix ci.
  Rachelint
  committed
  4c6799f
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  93e5f9d
- return error when found blocked approach not supported.
  Rachelint
  committed
  7542b49
Commits on Jan 21, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  kamillecao
  committed
  a0e467f
Commits on Jan 26, 2026
- fix compile.
  kamillecao
  committed
  53b8fa0
Commits on Feb 21, 2026
- fix compile.
  kamillecao
  committed
  78eec5a
Commits on Feb 23, 2026
- finish refactor of group_values_primitive.
  kamillecao
  committed
  aa808d6
Commits on Apr 6, 2026
- fix error macro.
  kamillecao
  committed
  f95384d
- fix NullState accumulate tests for blocked groups.
  
  kamillecao
  and
  claude
  committed
  6bda26e
- allow accs empty, and add some necessary asserts.
  kamillecao
  committed
  44ca669
- refactor: separate blocked groups eligibility check from side effects
  
  kamillecao
  and
  claude
  committed
  6be311c
- refactor: add SessionContextOptions for fuzz test config overrides
  
  kamillecao
  and
  claude
  committed
  8fae9fa
- docs: explain how test_blocked_groups_optimization forces the blocked path
  
  kamillecao
  and
  claude
  committed
  0e43431
Commits on Apr 14, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  056faaf
- fix: resolve compilation errors after merging main
  Rachelint
  committed
  0840671
- fix: correct collect_statistics default in information_schema test
  
  Rachelint
  and
  claude
  committed
  62483d4
Commits on Apr 15, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  7acfa7c
- fix: increase memory limits in spill tests for blocked groups
  
  Rachelint
  and
  claude
  committed
  e7fa863
- fix: split spill tests into flat/blocked variants
  
  Rachelint
  and
  claude
  committed
  b377457
Commits on Apr 18, 2026
- fix: avoid over-allocation in flat mode PrimitiveGroupsAccumulator
  
  Rachelint
  and
  claude
  committed
  fd2efd3
- fix: resolve clippy warnings in row_hash.rs
  
  Rachelint
  and
  claude
  committed
  fa0e639
- fix: resolve cargo fmt, clippy, and config docs issues
  
  Rachelint
  and
  claude
  committed
  46ec4b4
- fix: correct stale doc link to maybe_enable_blocked_groups
  
  Rachelint
  and
  claude
  committed
  cd51337
- refactor: organize aggregate tests and cover both flat/blocked modes
  
  Rachelint
  and
  claude
  committed
  14e6cf2
- rename things in blocks.rs to improve readable.
  Rachelint
  committed
  0069086
Commits on Apr 19, 2026
- refactor: replace VecDeque with Vec + start offset in Blocks
  
  Rachelint
  and
  claude
  committed
  8d05217
- rename `len` to `num_blocks`.
  Rachelint
  committed
  a68716b
Commits on Apr 26, 2026
- remove `Blocks` to replace `VecDeque` in `GroupValuesPrimitive`.
  Rachelint
  committed
  15f747d
- add back fast check path.
  Rachelint
  committed
  d30b2b9
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  c65a05d
Commits on Apr 28, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  authored
  a84d71c
- Fix blocked groups default regressions
  Rachelint
  committed
  4891fa5
- remove some useless codes.
  Rachelint
  committed
  9a7527d
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  authored
  a818cd8
Commits on Apr 29, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  902972f
- init with 128 cap.
  Rachelint
  committed
  47b6976
- Add block store abstraction
  Rachelint
  committed
  f0ba361
- Refactor group accumulator block stores
  Rachelint
  committed
  d1f96f1
- Encapsulate primitive group state
  Rachelint
  committed
  48078ec
Commits on May 3, 2026
- Encapsulate primitive group values state into enum adapter
  
  Rachelint
  and
  claude
  committed
  2e97173
Commits on May 5, 2026
- Move Blocks into block_store/blocked and simplify lifecycle
  
  Rachelint
  and
  claude
  committed
  066d379
Commits on Jun 5, 2026
- Simplify Block/BlockStore API by inlining block construction
  
  Rachelint
  and
  claude
  committed
  817c9d6
- Drop VecValues newtype in favor of blanket Block impl for Vec<T>
  
  Rachelint
  and
  claude
  committed
  37d5753
- Inline Block trait into block_store and co-locate emit impls
  
  Rachelint
  and
  claude
  committed
  f56db33
Commits on Jun 7, 2026
- Convert VecBlockStore from trait to struct wrapping BlockStore<Vec<T>>
  
  Rachelint
  and
  claude
  committed
  22167ae
- Convert SeenValueStore from trait to struct wrapping BlockStore<BooleanBlock>
  
  Rachelint
  and
  claude
  committed
  c51de4a
Commits on Jun 18, 2026
- refactor: add block store abstractions
  Rachelint
  committed
  3c51e0c
Commits on Jun 20, 2026
- Simplify null state construction
  Rachelint
  committed
  e49260f
- Derive null state block size from block store
  Rachelint
  committed
  2462069
Commits on Jun 21, 2026
- Remove BlockStore is_empty method
  Rachelint
  committed
  be352fd
- Import Deref traits in accumulate
  Rachelint
  committed
  8cd0092
Commits on Jun 22, 2026
- Merge branch 'main' into intermeidate-result-blocked-approach
  Rachelint
  committed
  d355609
- Fix primitive group values emit first test
  Rachelint
  committed
  eef162c
Commits on Jun 23, 2026
- Simplify blocked block store emission state
  Rachelint
  committed
  0b3179a
- Optimize primitive groups accumulator size
  Rachelint
  committed
  5869167

Commits

Commits on Apr 23, 2025

Commits on Apr 24, 2025

Commits on Apr 25, 2025

Commits on Apr 27, 2025

Commits on May 3, 2025

Commits on May 4, 2025

Commits on May 5, 2025

Commits on Jan 21, 2026

Commits on Jan 26, 2026

Commits on Feb 21, 2026

Commits on Feb 23, 2026

Commits on Apr 6, 2026

Commits on Apr 14, 2026

Commits on Apr 15, 2026

Commits on Apr 18, 2026

Commits on Apr 19, 2026

Commits on Apr 26, 2026

Commits on Apr 28, 2026

Commits on Apr 29, 2026

Commits on May 3, 2026

Commits on May 5, 2026

Commits on Jun 5, 2026

Commits on Jun 7, 2026

Commits on Jun 18, 2026

Commits on Jun 20, 2026

Commits on Jun 21, 2026

Commits on Jun 22, 2026

Commits on Jun 23, 2026