Search returns no results for non-Latin (Cyrillic/CJK/Greek) queries

[vitalibondar]

Search and filter return zero results for any query containing non-Latin characters — Cyrillic (Ukrainian, Russian), CJK, Greek, Arabic and so on — even when matching cards clearly exist. Latin queries work fine, so the failure is easy to mistake for "nothing matched".

Root cause

Search::Query#remove_invalid_search_characters sanitises input with terms.gsub(/[^\w"]/, " "). In Ruby (Onigmo) \w matches ASCII [a-zA-Z0-9_] only — it does not match Unicode letters. A non-Latin query is therefore reduced to whitespace, terms becomes blank, Search::Query fails its presence validation, and Search::Record.for_query falls through to else none — no results.

The FTS index itself is fine: non-Latin content is indexed and a raw MATCH finds it. The bug is purely in query sanitisation, which is why it affects both the SQLite and Trilogy adapters.

Suggested fix

Use the POSIX [[:word:]] class, which is Unicode-aware in Ruby:

- terms.gsub(/[^\w"]/, " ")
+ terms.gsub(/[^[[:word:]]"]/, " ")

The same ASCII-only \w appears in Search::Stemmer.stem, used on the Trilogy/MySQL path for both indexing and querying.

Status

I opened #2928 back in June with this change plus a Cyrillic test case mirroring the existing hyphenated-string test — it fails on main and passes with the change. I've since read CONTRIBUTING and understand a discussion is the right entry point, so I'm raising it here properly.

Happy to close the PR if you'd rather approach this differently, or to adjust it — the patch is two lines plus a test, and it currently merges cleanly.

For context on why this matters beyond my own use: any team writing card titles in a non-Latin script currently has search silently return nothing, with no error to explain why.

[0xshubhs]

Diagnosis is right and [[:word:]] is the correct swap. But the two paths aren't equally fixed by it, and I think that matters for how the PR gets reviewed.

On SQLite the two lines really are the whole fix. Search::Record::SQLite never stems on write, after_save :upsert_to_fts5_table puts the raw title and content into FTS5, and matching hands the query straight through:

scope :matching, ->(query, account_id) {
  joins("INNER JOIN search_records_fts ON search_records_fts.rowid = #{table_name}.id")
    .where("search_records_fts MATCH ?", query)
}

The migration builds the table with tokenize='porter', and FTS5's porter tokenizer is a wrapper around unicode61, which is Unicode-aware. So the index genuinely is fine, exactly as you found, and Search::Query is the only thing standing in the way.

Trilogy is a different story, because the stemmer runs at write time there. Search::Record::Trilogy:8:

before_save :set_account_key, :stem_content

def stem_content
  self.title = Search::Stemmer.stem(title) if title_changed?
  self.content = Search::Stemmer.stem(content) if content_changed?
end

So the same ASCII \w in Search::Stemmer.stem has already destroyed the non-Latin text in the stored title and content columns of every existing row. Fixing both regexes fixes new writes and new queries, and existing rows stay empty of anything non-Latin. Search keeps returning nothing on MySQL and it looks like the patch didn't work. Any deployment on Trilogy needs a full re-stem of the search_records_* shards as part of the change, and your test won't surface that because it creates its records after the fix is in effect.

The other thing I'd separate out is CJK, because the regex doesn't fix it on MySQL at all. The fulltext indexes are plain:

t.index [:content, :title], type: :fulltext

with no WITH PARSER ngram, in both 20251113190256_create_search_record_shards.rb:22 and 20251121112416_remove_old_fulltext_indexes_from_search_records.rb:14. MySQL's default parser finds token boundaries with whitespace and punctuation, so Cyrillic, Greek and Arabic tokenize correctly once you stop stripping them, but a Chinese or Japanese sentence arrives as a single token and substring queries still miss. That needs the ngram parser and a rebuild, which is a much bigger change than two lines.

None of that argues against #2928, it's still the right first move. It would just land better as "fixes space-delimited non-Latin scripts on SQLite, plus a reindex for Trilogy, CJK on MySQL tracked separately" than as a general non-Latin fix, since the current framing promises more than the patch can deliver on the MySQL path.