[enhancement] CJK (Chinese/Japanese/Korean) full-text search broken — hardcoded unicode61 tokenizer doesn't segment CJK

Environment: basic-memory 0.22.1, SQLite 3.43.2

Problem
The FTS index hardcodes tokenize='unicode61 tokenchars 0x2F' with prefix='1,2,3,4' (in basic_memory/models/search.py:91 and the migration alembic/versions/cc7172b46608_update_search_index_schema.py). unicode61 performs no CJK word segmentation, so a continuous run of CJK characters becomes a single token, and only prefixes match.

Repro: a note contains 即适者生存，生存等于正确的道理:

search-notes "适者生存" → no results (适者生存 is a mid-token substring of 即适者生存, not a prefix).
search-notes "财务护城河" in a note containing 做好财务护城河 → wrong/no result, same cause.

This makes keyword search effectively unusable for CJK notes.

Why trigram is not sufficient
SQLite's built-in trigram tokenizer only matches queries ≥3 characters. CJK words are frequently 2 characters (止损, 健康, 红线), so switching to trigram would break the most common search pattern. (Verified: trigram returns nothing for 2-char queries.)

Request
Add CJK word segmentation support, e.g.:

A config option for the FTS tokenizer (so users can choose trigram or a jieba-backed tokenizer), and/or
Built-in jieba segmentation (pre-segment CJK before indexing and before querying).

Workaround
None clean. CJK users must rely on vector/semantic search for phrase lookup.

I timeboxed the fallback investigation for v0.23.

The report is valid, but there is not a small bounded fallback on top of the current index:

FTS5 prefix matching cannot recover a mid-token CJK phrase from a unicode61 token.
A LIKE fallback would scan stored note content whenever FTS misses.
An fts5vocab substring fallback would scan the token vocabulary for the same reason; there is no usable prefix bound when the match starts in the middle of a token.
The built-in trigram tokenizer needs a separate/rebuilt index and still does not cover common two-character CJK queries.
Pre-segmentation or a configurable tokenizer changes both indexing and query preparation, requires an index rebuild/migration story, and needs a cross-CJK design rather than a Chinese-only assumption.

That crosses the agreed v0.23 stop line (new index/tokenizer subsystem or unbounded query-time scan), so I am punting this from v0.23 and leaving the issue open for a properly designed search-index change. Semantic/hybrid search remains the current workaround; it does not make the keyword-search bug resolved.

Verified, and the diagnosis is right for both backends:

SQLite: tokenize='unicode61 tokenchars 0x2F' (models/search.py, migration cc7172b46608), query prep only adds prefix wildcards. On the bundled SQLite 3.53.3 with your sample text: 适者生存, 财务护城河, 生存 → 0 hits; 止损 (a standalone run) → 1. With the trigram tokenizer every ≥3-char query hits, but 2-char queries (止损, 生存) → 0 — your objection to trigram stands.
Postgres: to_tsvector('english', …) / to_tsquery('english', …) throughout; PG's parser also emits a CJK run as a single token, and switching to simple only drops stemming. pg_trgm is enabled but only for title/permalink link resolution, with the same ≥3-char floor.

What we'll do:

Now: document that keyword search matches whole CJK runs only and that semantic/hybrid search with a multilingual model (jinaai/jina-embeddings-v2-base-zh, multilingual-e5-large, both already in docs/semantic-search.md) is the working path today — that note is missing.
Next: a search_fts_tokenizer setting (unicode61 | trigram) applied in the DDL with a versioned FTS rebuild through bm reindex; trigram covers all CJK scripts with no new dependency and we document the 2-char floor. Postgres parity (a pg_trgm GIN on search_index content) is a separate change.
jieba/ICU pre-segmentation stays off the table for now — Chinese-only, new dependency, and query-time and index-time have to agree.

docs(core): drop the broken star-history chart and document CJK keywo…

Scope decision after reviewing the cross-backend and Cloud requirements:

This issue is FTS only. It covers exact lexical retrieval for CJK/no-whitespace text on both SQLite FTS5 and PostgreSQL full-text search.
It will not change embedding providers, models, dimensions, vector storage, or semantic chunking.
The implementation should use one shared application-level document/query analysis contract so SQLite and PostgreSQL agree. We should not expose SQLite tokenizer names as the product abstraction.
User-facing naming should describe mechanisms rather than call the existing behavior “standard”; the working vocabulary is the existing word_terms channel plus an additive script_ngrams channel.
Existing rows will require a normal FTS rebuild (bm reindex locally and a fleet reindex in Cloud). This is separate from bm reindex --embeddings.

The earlier comment's proposed search_fts_tokenizer = unicode61 | trigram direction is superseded by this cross-backend scope. Trigram's two-character floor remains disqualifying for the reported examples.

Multilingual FastEmbed model selection and the Cloud FastEmbed → PostgreSQL/pgvector migration are now tracked separately in basicmachines-co/basic-memory-cloud#2409 so we can benchmark retrieval quality, English regression, resource use, pgvector behavior, and fleet cutover before changing any embedding default.

Navigation Menu

Repository navigation

[enhancement] CJK (Chinese/Japanese/Korean) full-text search broken — hardcoded unicode61 tokenizer doesn't segment CJK #1294

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

[enhancement] CJK (Chinese/Japanese/Korean) full-text search broken — hardcoded unicode61 tokenizer doesn't segment CJK #1294

Description

Activity

phernandez commented on Aug 23, 2026

phernandez commented on Aug 29, 2026

phernandez commented on Aug 29, 2026

Metadata

Metadata

Assignees

Labels

Projects

Milestone

Relationships

Development

Issue actions