[elee1766]

Dependency and status

Stacked follow-up to #1995. Do not merge before that storage PR. Only changes after parent commit 071454bff belong to this experiment. This remains a draft: disk usage, preparation cost and broad-query latency are material tradeoffs.

Changes

• Apply picker visibility/debug budgets in SQL before fetching summary JSON.
• Maintain optional FTS5 trigram indexes in the same transactions as persisted-row writes.
• Add explicit offline preparation with compact (detail=none), column and positional modes. Positional remains the default. Normal open/list/save/search never builds or replaces an index.
• Compact and column use indexed LIKE with content verification. Positional uses phrase postings. Short and Unicode queries retain the existing fallback.
• Preserve file storage, JSON import/export, scoring and exhaustive search.
• Add public-interface benchmarks and a six-mode same-row comparison. No private content or query strings are published.

Index alternatives

Sample: 64 size-quantile real sessions, 12,651 rows, 75.8 MB payload, SQLite 3.46.0. Fixed-order screening with three repeats, not a randomized throughput claim.

Index	Additional bytes	Substring candidate median	Exact candidates across all six queries
None	0	162.5 ms	Yes
B-tree	90,255,360	157.6 ms	Yes
Word FTS	40,095,744	0.06 ms	No
Compact trigram	17,956,864	20.9 ms	Yes
Column trigram	44,380,160	24.0 ms	Yes
Positional trigram	167,825,408	0.44 ms	Yes

Word FTS omitted all eight substring-matching sessions and nine short-query matches. Its faster times are not equivalent work. Compact used 55% less index space than word FTS and 89% less than positional in the sample. Column was not a clear improvement over compact. Small-row update timings are recorded but are insufficient for a throughput conclusion.

Full corpus: 3,018 sessions. Existing compact and positional databases occupy 4.71 GB versus 11.01 GB, a 57% total-space reduction. Those earlier-built indexes retain token-count tables; new builds use columnsize=0.

Three alternating fresh-process public SessionSearchTool::execute runs returned identical result digests in all six categories. Compact was generally slower: selective medians 6.55 versus 3.46 seconds, substring 6.35 versus 3.91 seconds, multi-term 8.84 versus 4.02 seconds, broad 9.56 versus 2.23 seconds (compact versus positional). Timing variance was high, so these are local tradeoff observations, not portable speed guarantees. Defaults remain unchanged.

Earlier baseline comparison

Before versus positional: picker 727.8 -> 287.7 ms with the same 800 rows, absent 1272.9 -> 70.8 ms, selective 1661.2 -> 1551.0 ms, broad 996.4 -> 1449.5 ms. Initial positional preparation took 20m38s and grew the database from 3.48 to 11.01 GB. No compression was added.

Two bounded searches differed from the original whole-token Bloom prefilter. Candidate auditing found that Bloom omitted 475/491 substring matches and 35/224 multi-term matches; FTS matched raw candidate sets exactly. Exhaustive public outputs matched across revisions. Bounded scoring still means indexed and exhaustive outputs need not agree.

Validation

• Current changes: 11 storage tests and 28 search tests pass, one existing benchmark ignored.
• Compact/column mode-switch regressions cover substring matching, noncontiguous false positives, append and truncation.
• Full-corpus compact versus positional public output parity passed all six categories, three runs each.
• Earlier SQLite picker tests: four passed. No full-workspace green claim.
• No live configuration, daemon, installed binary or session store changed.

Full methodology, limitations and aggregate JSON are under docs/SESSION_SQLITE_SEARCH.md and docs/benchmarks/. Maintained regressions and reproducible benchmark examples are retained; temporary prototypes are not included.