Skip to content

perf(table): search global index shards concurrently#589

Merged
XiaoHongbo-Hope merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:optimize_global_index_shard_concurrency
Jul 22, 2026
Merged

perf(table): search global index shards concurrently#589
XiaoHongbo-Hope merged 1 commit into
apache:mainfrom
XiaoHongbo-Hope:optimize_global_index_shard_concurrency

Conversation

@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor

Summary

Search independent sorted global-index shards concurrently instead of reading BTree and bitmap shards one by one. The existing global-index.thread-num option now bounds per-scan shard I/O, while a value of 1 preserves sequential execution.

Changes

  • Complete file pruning and unsupported-fallback decisions before starting shard I/O, so a later unsupported shard still falls back without racing an earlier read error.
  • Query independent BTree and bitmap shards with bounded concurrency and fold their row-id bitmaps incrementally into the final result.
  • Keep predicates within each shard sequential so its reader is opened once, and return successful BTree readers to the existing cache.
  • Parse global-index.thread-num only when a data-evolution scan will actually evaluate global-index predicates.
  • Include index kind and file name in shard query errors.

Performance

Measured against the same object-store snapshot with identical result signatures:

global-index.thread-num Plan time
1 11.809s
8 2.790s
32 1.944s
64 1.810s

Testing

  • cargo test --locked -p paimon --lib table::global_index_scanner::tests (39 passed)
  • cargo test --locked -p paimon --lib (1801 passed, 1 ignored)
  • cargo fmt --all -- --check
  • cargo clippy --locked -p paimon --all-targets -- -D warnings

The new tests verify both sequential/concurrent limits and that fallback preflight completes before shard I/O.

Notes

This reuses global-index.thread-num, introduced for primary-key vector search in #556. It does not depend on #586 or #587.

@QuakeWang QuakeWang left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@XiaoHongbo-Hope
XiaoHongbo-Hope merged commit 2ef0702 into apache:main Jul 22, 2026
25 of 26 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants