Vector/hybrid filter intersection drops candidates beyond VECTOR_FILTER_SCAN_LIMIT #1431

Summary

When a vector or hybrid query carries a structured filter, the filter-only FTS call is capped at VECTOR_FILTER_SCAN_LIMIT (50,000) rows, and every vector candidate outside that page is treated as disallowed. A highly similar, genuinely matching result can therefore vanish purely because of where it fell in that scan.

Raised during review of #1426 (SPEC-82 temporal search) and declined there on scope — this is pre-existing and not temporal-specific. Filing so it isn't lost.

Why it is not merely "arbitrary"

With search_text=None there is no relevance signal in the ordering, so the retained 50,000 are not a random sample:

• Postgres tiebreaks search_index.id ASC — the survivors are deterministically the earliest-indexed rows, i.e. systematically biased against newer content
• SQLite has no id tiebreaker at all

Why it is not covered by the eventual-consistency carve-out

CLAUDE.md asks reviewers not to raise derived-state races that converge on a later write or index pass. This is not one of those. It is a result-completeness bug: the query returns a wrong answer now and returns the same wrong answer on every rerun. Nothing repairs it.

Scope — this is not about temporal

The filter_requested block, the capped self.search(...) call, and the allowed_keys intersection are byte-identical on posix-meta-predicates, the base branch. VECTOR_FILTER_SCAN_LIMIT dates to #550, with #1141 on top. SPEC-82 adds temporal to a list that already held nine filters with the identical exposure.

temporal is in fact the least reachable of the ten — it only matches rows carrying authored valid-time markup, whereas search_item_types=[observation] or a shallow file_path_prefix crosses 50k on any large vault. Fixing it for the newest and narrowest filter alone would leave the eight more reachable ones broken while making the path look safe, which is worse than leaving it visibly unfixed.

Why the obvious fixes are not small

• Intersect with the candidate keys (the right design): search() has no id-restriction parameter, so this means a new argument on the abstract signature plus both backend implementations plus batched IN lists — the file already batches hydration at 250 because both engines cap bind parameters.
• Exhaust the pages: removes the safety valve that stops a filter-only scan materializing an entire project's search index into memory.
• Raise the constant: moves the cliff, does not remove it.

Suggested direction

Scope a fix to the whole filter-intersection path rather than one filter, with candidate-key restriction as the design: push the candidate ids into the filter query instead of paging the filter and intersecting client-side.

Reachability

Requires a project whose filter-matching rows exceed 50,000 — large vaults and cloud tenants, not typical local use. That is what makes it a tracked issue rather than a release blocker.