Repository navigation
Description
Problem
basic-memory reindex --search cannot repair database-only divergence: it selects files to
index purely from file-content checksum changes, so when the observation/relation/
search_index tables disagree with an unchanged markdown file, the incremental pass selects
zero files and reports success while search keeps serving stale rows.
Field report on #1214 (SQLite, concurrent MCP clients): after trimming duplicate
observation rows, search_index still held 3886 rows of type='observation' against 2146
observation rows. reindex --search reported Indexing files... 0/0 files and changed
nothing; until reindex --full --search ran, search answered with content no longer in the
store.
Root cause
select_project_index_target_files (src/basic_memory/indexing/project_index_coordinator.py:229-242)
builds its target set from change_report.new_files | modified_files, which
change_detector.py derives purely from file checksums (load_indexed_file_checksums).
DB-only divergence leaves every file in unchanged_files, so nothing reaches
search_service.index_entity_data — the only code path that deletes and rewrites an
entity's observation search rows. _purge_stale_search_rows only handles rows for deleted
entities, and only runs in the --embeddings path.
Proposed direction
Add a DB↔FTS reconciliation sweep to the search reindex path (extend
_purge_stale_search_rows or a sibling): purge search_index rows whose id no longer
exists in the owning table (type='observation' → observation.id, type='relation' →
relation.id), and optionally detect entities whose row counts disagree and re-enqueue just
those entities, so --search without --full converges without a full rebuild.
The one-time damage from the observation duplication bug is handled by the #1214 migration's
FTS orphan purge; this issue is about making incremental reindex --search able to repair
such divergence in general.
Acceptance criteria
-
reindex --search(no--full) removessearch_indexrows oftype='observation'/type='relation'whose backing row no longer exists. - A seeded DB↔FTS divergence on an unchanged file converges via
reindex --searchalone (regression test). - No full-file re-parse is required for the purge path;
--fullbehavior unchanged.
Activity
claude commentedon Aug 11, 2026
Claude finished @phernandez's task in 41s —— View job
Triage
Type: Bug (confirmed —
buglabel already applied)Component: Database/sync —
reindex --searchindexing pipeline (src/basic_memory/indexing/project_index_coordinator.py); not cloud-specificPriority: High — search silently serves stale/incorrect content after DB-only divergence, and the documented incremental repair path (
reindex --search) reports success while doing nothing. No data loss, but a correctness/trust issue for search results.Complexity: Medium — the fix is scoped (extend the FTS reconciliation sweep already present for the
--embeddings/deleted-entity path to also run for--search, purge orphanedsearch_indexrows by id, and add a regression test for a seeded DB↔FTS divergence on an unchanged file). No architectural changes required, but touches the_purge_stale_search_rows/search_service.index_entity_datainteraction so needs careful testing to avoid affecting the unrelated--fullpath.Verification: Read
select_project_index_target_filesinsrc/basic_memory/indexing/project_index_coordinator.py:229-242— it does build its target set solely fromchange_report.new_files | modified_files, which come from file-checksum diffing (change_detector.py), so a DB-only divergence (no file change) indeed produces an empty target set andreindex --searchno-ops. Root cause as described in the issue is accurate.Duplicate check: No existing open or closed issue duplicates this. #1214 is a related companion issue (generation-versioned observation persistence / the field report that surfaced this), not a duplicate — already correctly cross-referenced in the issue body.
Labels applied:
bug, no cloud-specific component.