Skip to content

[core][python] Fix index-specific search mode defaults - #8844

Merged
JingsongLi merged 10 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/scalar-global-index-search-mode-default
Jul 26, 2026
Merged

[core][python] Fix index-specific search mode defaults#8844
JingsongLi merged 10 commits into
apache:masterfrom
XiaoHongbo-Hope:fix/scalar-global-index-search-mode-default

Conversation

@XiaoHongbo-Hope

@XiaoHongbo-Hope XiaoHongbo-Hope commented Jul 24, 2026

Copy link
Copy Markdown
Contributor

Purpose

  • Add index-specific search modes in Java and Python.
  • Default scalar to full, vector and full-text to fast.
  • Keep global-index.search-mode as an optional legacy fallback.
  • Limit scalar fallback to vector-index coverage in vector fast mode.
  • Keep partition-only vector filters out of scalar prefiltering.
  • Force predicate mutations to scan scalar indexes in full mode.

Resolution order: explicit family option, explicit global option, family default.

Tests

  • Python search-mode/vector regressions: 181 passed, 14 subtests.
  • Java: CoreOptions, BTree, vector, and full-text tests passed; VectorSearchBuilderTest 37/37.
  • Flink DELETE/MERGE data-evolution IT classes: 57/57.
  • Maven package, Spark common package, Flake8, and diff checks passed.

@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the fix/scalar-global-index-search-mode-default branch 5 times, most recently from 4eefdda to fb38ba0 Compare July 24, 2026 11:53
…nt row loss

Scalar (sorted/bitmap) global-index queries used the vector/full-text
`global-index.search-mode` = fast, which returns no unindexed ranges, so rows
committed but not yet covered by the index were pruned before being read and
`WHERE <indexed_col> = X` silently returned incomplete results.

- Add `scalar-index.search-mode` (default `full`). Resolution priority:
  explicit `scalar-index.search-mode` -> an explicit `global-index.search-mode`
  (inherited) -> default `full`.
- Thread an optional `search_mode` through DataEvolutionGlobalIndexCoverage
  and DataEvolutionGlobalIndexScanner.unindexed_rows.
- Only the pure scalar scan (file_scanner) passes the scalar mode. Vector /
  full-text callers leave it unset and keep the fast default, so a vector scan
  with a scalar filter is not widened into a full-table brute-force scan.
@XiaoHongbo-Hope
XiaoHongbo-Hope force-pushed the fix/scalar-global-index-search-mode-default branch from fb38ba0 to ce3071e Compare July 25, 2026 14:25
@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Fix/scalar global index search mode default [python] Default scalar global-index search mode to full Jul 25, 2026
@XiaoHongbo-Hope
XiaoHongbo-Hope marked this pull request as ready for review July 26, 2026 02:41
@JingsongLi

Copy link
Copy Markdown
Contributor

We should modify Java too. And we need to modify the default value of global-index.search-mode to none. Introduce vector-index.search-mode and full-text-index.search-mode too, their default value should be fast.

@XiaoHongbo-Hope XiaoHongbo-Hope changed the title [python] Default scalar global-index search mode to full [core][python] Fix index-specific search mode defaults Jul 26, 2026
@XiaoHongbo-Hope

Copy link
Copy Markdown
Contributor Author

We should modify Java too. And we need to modify the default value of global-index.search-mode to none. Introduce vector-index.search-mode and full-text-index.search-mode too, their default value should be fast.

Updated

@JingsongLi JingsongLi left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

+1

@JingsongLi
JingsongLi merged commit 3f84bbd into apache:master Jul 26, 2026
19 of 21 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants