Repository navigation
Retrieval pipeline improvements: reranking, length normalization, noise filtering, adaptive recall #666
Description
Context
Analyzed memory-lancedb-pro (LanceDB-based OpenClaw memory plugin) and their demo. While architecturally different from BM (opaque vectors vs human-readable plain text), their retrieval pipeline has several techniques worth adopting.
Our LoCoMo benchmark baseline: Recall@5 76.4%, Recall@10 85.5%, MRR 0.658. Weakest areas: single_hop 57.7%, temporal 59.1%. Root cause identified in #577: RRF scoring flattens results, FTS outperforms vector for observations.
Proposed Improvements
1. Cross-encoder reranking (highest impact)
After initial hybrid retrieval (vector + FTS), run a second pass through a cross-encoder reranker (e.g., Jina reranker-v3, Voyage rerank-2.5). This would re-score candidates based on query-document semantic relevance rather than just embedding similarity.
memory-lancedb-pro uses: 60% cross-encoder score + 40% original fused score, with graceful fallback to cosine similarity on API failure.
Implementation options:
- Cloud API: Jina, Voyage, Pinecone (cheap per-query cost)
- Local: BAAI/bge-reranker via Ollama-compatible endpoint
- Config-driven: optional, off by default, provider-agnostic
This directly addresses #577 (RRF flattening) and should significantly improve Recall@5 and MRR.
2. Length normalization
Long notes/observations currently have an outsized influence on search results. A length normalization step (anchor around 500 chars) would prevent verbose entries from dominating over precise, short observations.
3. Noise filtering / adaptive retrieval
Skip memory retrieval entirely for queries that don't need it: greetings, slash commands, simple confirmations, emoji-only messages. Also filter low-quality content from capture: agent refusals, meta-questions, boilerplate.
This reduces wasted retrieval cycles and keeps the index cleaner.
4. MMR diversity filtering
When multiple results are very similar (cosine > 0.85), demote duplicates to increase result diversity. Prevents near-duplicate observations from consuming all top-K slots.
What we already do better
- Human-readable plain text (their memories are opaque vector rows)
- Knowledge graph with relational structure (observations + relations)
- Bidirectional editing (humans can correct memories by editing files)
- Schema system for structured note types
- Git history as provenance trail
Priority
- Cross-encoder reranking (biggest recall improvement per effort)
- Length normalization (quick win)
- MMR diversity (moderate effort)
- Noise filtering / adaptive retrieval (nice to have)
References
- memory-lancedb-pro: https://github.com/CortexReach/memory-lancedb-pro
- LanceDB demo: https://github.com/lancedb/openclaw-lancedb-demo
- LoCoMo benchmark results: #608
- RRF scoring issue: BUG: Hybrid RRF fusion dilutes vector scores with weak FTS scores #577
Activity
claude commentedon Mar 14, 2026
Claude finished @bm-clawd's task in 1m 34s —— View job
Triage Analysis
Classification
Labels applied:
enhancementDuplicate / Related Issues
Partial overlap with #618 ("Add reranking step to search pipeline (local cross-encoder)"), which covers cross-encoder reranking in detail with a tiered approach (Cloud / local OpenAI-compatible API / no-key fallback). Issue #666 is broader in scope, covering:
Recommend either closing #618 as a sub-item of this issue, or keeping #618 open and cross-linking them.
Triage Notes
Item 1 — Cross-encoder reranking is well-specified and aligns with the existing #618 proposal. The config-driven, provider-agnostic design is the right approach. Key decision: whether to ship cloud-only, local-only, or both in a first pass.
Item 2 — Length normalization is a quick win. BM25/FTS already has some length normalization built in (
bparameter), but applying it post-fusion to the combined score is straightforward. Anchoring around 500 chars is a reasonable heuristic to validate against the LoCoMo benchmark.Item 3 — Noise filtering / adaptive retrieval (skip retrieval for greetings, slash commands, etc.) is useful but should be implemented carefully to avoid false negatives — short queries like "what did we discuss?" should still trigger retrieval.
Item 4 — MMR diversity filtering (demote near-duplicates with cosine > 0.85) is a well-understood technique and fits naturally as a post-reranking step.
Suggested implementation order (matching the issue's priority list):
phernandez commentedon Aug 11, 2026
Closing: the priority-1 item — cross-encoder reranking, "biggest recall improvement per
effort" — shipped in #1143 (FastEmbed local + LiteLLM hosted providers, opt-in config),
with phase-1 coverage hardening in #1232 and the remaining verification tracked in #1231.
Two deliberate design divergences from this issue's sketch, for the record:
replace the fused scores for a fixed candidate prefix, with rank-only demoted tail
scores — required for stable pagination across pages.
because a silently degraded page could duplicate or omit results relative to an
already-served reranked page.
Items 2-4 (length normalization, noise filtering / adaptive retrieval, MMR diversity) are
intentionally not carried forward here: the reranker changes the retrieval baseline they
were proposed against, and #1231 Phase 3 builds the quality-regression harness (golden
queries, hit@k/MRR, LoCoMo re-run) to measure what remains. If post-rerank numbers still
show the verbose-note dominance or duplicate-crowding patterns, open fresh targeted issues
against the new baseline. Entity-aware candidate improvement continues in #951.