Domain-adaptive fine-tune as orthogonal R@5 lift on top of MemPal raw #1249
Replies: 6 comments 1 reply
-
|
Congrats on v3.3.4 — the DB size reduction is impressive. Quick question: did the storage optimisation affect the index structure at all, or is the |
Beta Was this translation helpful? Give feedback.
-
|
I'm reading your work! Excited to learn more. |
Beta Was this translation helpful? Give feedback.
-
|
Quick follow-up on the May 1 question about v3.3.4+ protocol equivalence, I re-ran all three rows on v3.3.5 (latest release as of today) and also did a controlled v3.3.3 repro to isolate the source of any movement. Numbers below. Three runs on v3.3.5 (full 500q, matched protocol)Same
Three takeaways
What the deltas mean
Encoder fine-tune and hybrid retrieval are still adding lift on top of each other at v3.3.5. R@5 is ceiling-bounded (close to 1.000), so R@1 is the honest comparison and the orthogonality reads clearly there. Reproducecd ~/Projects/mempalace && git checkout v3.3.5
cd ~/Projects/adaptmem
PYTHONPATH=/path/to/mempalace python benchmarks/mempal_bench_with_ft.py \
--bench-script /path/to/mempalace/benchmarks/longmemeval_bench.py \
--data-file /path/to/longmemeval_s_cleaned.json \
--ft-model /path/to/minilm-lme-ft-300 \
--mode {raw|hybrid_v4} \
--out results.jsonlThe three v3.3.5 result JSONLs are committed in If hybrid_v4 reruns on top of these numbers are useful to compare against your own internal measurements, happy to share the result JSONLs directly. Otherwise this is just to close the May 1 question with current numbers. |
Beta Was this translation helpful? Give feedback.
-
|
Quick update on the v3.3.5 rerun comment, running on the same matched-protocol harness, the ft-v4 encoder upgrade plus a three-stage rerank stack pushes the R@1 0.95 row to R@1 0.99 (5 fails / 500). Stages on top of
Remaining 5 fails decompose as 1 abstain ( Repo: nakata-app/adaptmem, Two possible integration shapes if interesting: an opt-in Happy to share JSONL artefacts and pipeline scripts under whichever direction fits. |
Beta Was this translation helpful? Give feedback.
-
|
jphein, Önceki yanıt için teşekkürler. 20 probe'luk ablation üzerinde paired bootstrap (10K resample, 95% CI) koşturdum, iki tarafın da görmesi için sayıları aşağı koyuyorum. B vs A (heading-aware vs paragraph), bizim corpus ve probe set:
Her tek probe için rank birebir aynı çıkıyor. Paragraph ve heading-aware aynı drawer parçalanışı üretiyor (3759 vs 3747 chunk @ cs=400). Yani bizim probe set'inde markdown heading ayrımı "ateşlemiyor". Kavramsal argümanın yanlış demiyorum, ölçemiyorum. C vs A (AST vs paragraph), senin "complexity without lift" tavsiyenin tersi:
cs=800'de AST, iki encoder ile de 95% CI sıfırın üzerinde lift veriyor. cs=400'de kayboluyor. Talep: Bizim probe set 20 entry hard-coded (
Probe YAML'ı (script'in Code için "structured extraction + graph traversal" yaklaşımının yazısı yayında mı? Pipeline'ı yazıya görmek isterim, bizim retrieval surface'inde paralel bir track yararlı olabilir. teşekkürler, |
Beta Was this translation helpful? Give feedback.
-
|
@nakata-app — thanks for running the paired bootstrap with the CIs; the B-vs-A flat reading and the C-vs-A cs=800 lift on your 20-probe set both look defensible at the n you ran. Quick reply to your three asks, plus a cross-reference that may compose with the additive-axes story. The n=200 probe setLives on the fork at Shape: 200 questions, file-shaped The YAML is self-contained — no On the "structured extraction + graph traversal" questionThe "skip chunking for code, do AST-extraction-into-graph" framing in this thread came from @xg-gh-25 on #1384, not from us — worth attributing there. That said, the parallel-track angle is reasonable because our fork is doing graph traversal at the substrate layer, just from a different starting point:
So we have the graph traversal substrate but not the AST-to-graph extraction step. xg-gh-25's pipeline note suggests the missing piece is upstream of the graph, not in it. Worth their own writeup; I'll let them speak to that. FT-300 independent reproduction (just landed)Cross-reference your additive-axes story directly: reproduced FT-300 end-to-end on katana this morning from
Same on 500q full (training questions included): R@5 = 0.9980 (5/6 categories saturate at 1.000; small dip on single-session-assistant at 0.9821). Wall clock 56s train + 18s test on the GPU. Reproduces inside published noise — your FT-300 protocol is portable. Full writeup + reproducible split JSON: For methodological completeness — three code-tuned variants from your Composition direction worth checking nextYour matched-protocol numbers had hybrid_v4 + FT-300 + 3-stage rerank at R@1 = 0.99 in SPRINT_4_FINAL.md. Substrate-floor parity in our SME #9 thread confirms postgres+pgvector + MiniLM = chromadb + MiniLM byte-identically (R@5 = 0.9660, per-category exact match across all 6 qtypes). So in principle the FT-300 + hybrid_v4 + rerank stack should compose into our postgres substrate the same way it composes into upstream chromadb. We haven't measured that yet — the hybrid retrieval layer on the postgres backend is the next item that needs an SME-side reading. Will post when that lands. Question back: your 🫏 |
Beta Was this translation helpful? Give feedback.
-
|
@nakata-app — wanted to follow your additive-axes story with a cross-domain data point that I think extends it rather than challenges it. Short version: your in-domain lift reproduces and holds up for us; when we carried the same encoder to a different corpus it flattened; and a finding from our side suggests there may be a fourth orthogonal layer worth stacking on top. First — the in-domain lift is real, and it reproducesYour orthogonal-layers framing is compelling, and the numbers back it. Your published table has MemPal raw R@5 0.966 → +FT-300 0.980 → +hybrid_v4+FT-300 0.990, with R@1 climbing 0.806 → 0.862 → 0.916 — encoder fine-tune and hybrid retrieval each adding lift on top of the other. We reproduced the FT-300 leg end-to-end on our own hardware (katana, fresh seed=42 300/200 split) and the held-out 200q test hit R@5 = 1.000 (R@1 0.925) — inside your published noise. So the in-domain encoder lift isn't a one-machine artifact; the protocol is portable and the R@5 lift toward ceiling is genuine. No argument from us there. Where it gets interesting — a cross-domain transfer testWe then did something your thread hadn't covered: carried the same FT-300 encoder to a deliberately different corpus — jp-realm-v0.1, a 30-question probe set over a personal technical knowledge base (135k drawers of code, infra notes, RFCs), scored by substring
Here the lift didn't transfer: R@5 0.5172 → 0.5172, flat. 24 of 29 covered questions move exactly 0.0 — the FT encoder ranks the same drawers as base. A from-recipe re-train of the fine-tune (third leg) landed within ±2pp of base too, so two independently trained FT encoders both no-op'd on this corpus. (One honest detail: the published FT-300 we have carries code/scientific-computing training content, so against a personal technical KB it's genuinely out-of-domain — the cleanest version of the test.) Read together with your numbers, this is completely consistent if the lift is domain-specific: strong when the fine-tune corpus and eval corpus are the same family, flat across a corpus shift. That's not a knock on the method — it's a boundary on it. So the real question back, collaborator-to-collaborator: have you seen the orthogonal lift hold across a corpus shift, or does it want hard-negative re-mining on the target corpus to travel? Your A possible fourth orthogonal layerOne more finding that I think composes with your encoder+hybrid stack rather than competing with it. On oracle LongMemEval — gold session pinned in context, retrieval held at its 0.974 R@5 ceiling — we measured reader QA at only ~50%: a ~45pp R@5→QA gap (our #116). The right evidence is in front of the reader and it still misses. So on that corpus an encoder lift driving R@5 from 0.966 toward 1.000 is real but doesn't, on its own, move end-to-end QA — the bottleneck has shifted downstream to the reader/consumption layer. On top of your stack — encoder-FT, hybrid retrieval, the rerank cascade — this reads like one more orthogonal layer: reader/prompt design. (Our stratified n=150 retrieval A/B also had graph/age fusion ~neutral — R@5 92.67% vs 92.00% — which is why we're now spending our attention on the reader rather than retrieval.) Full writeups and the convergent findings are on our results page: https://techempower-org.github.io/multipass-structural-memory-eval/site/#benchmarks Genuinely — the reproducibility of your protocol is what let us run the cross-domain test at all. Curious to hear if corpus-shift transfer is something you've poked at. 🫏 |
Beta Was this translation helpful? Give feedback.
Uh oh!
There was an error while loading. Please reload this page.
-
Hi MemPal team,
We've been using LongMemEval to evaluate a small open-source library
called
adaptmem, a 200-line hard-negative mining + contrastive fine-tune wrapper around
SentenceTransformers, and the numbers we got line up cleanly with
the work you've already published. Wanted to share back, see if
it's interesting.
What we measured
Same dataset (
longmemeval_s_cleaned.json), same encoder family(MiniLM-L6, ~90MB), run through your own
longmemeval_bench.py(monkey-patched to swap the encoder, zero changes to your eval logic).
Only the fine-tune step differs.
Three findings worth flagging:
Raw baseline R@5 = 0.966 matches your published number exactly.
Independent confirmation that your protocol is fully reproducible,
we didn't need any hints beyond the repo README.
FT-300 + raw mode: +5.6pt R@1, +1.4pt R@5. R@1 is where
contrastive fine-tuning moves the needle most, the model learns to
rank the right session first, not just in top-5.
FT-300 + hybrid_v4: +11pt R@1, +2.4pt R@5. Fine-tune and
hybrid retrieval stack orthogonally, each adds lift on top of the
other.
Possible integration shape
If interesting, a
mempal-adaptintegration could look like:"adapter": before ingestion, point adaptmem at the labelled-query
set (if available), it produces a domain-tuned encoder that mempal
then uses for embedding.
config load time.
We don't have strong feelings about the shape, happy to defer to
your design preferences. The point of this thread is just to put
the numbers in front of you and see whether there's a productive
conversation here.
Reproduce
Three committed result JSONs in
benchmarks/:results_minilm_baseline_400.json, raw protocol confirmation.results_ft100_400.json, self-contained FT-100 reproduce.results_ft300_direct.json, FT-300 reference run.Either outcome is fine
If this isn't a fit for mempal's direction, no problem, adaptmem
will keep on as a standalone tool. Just thought it was worth showing
the numbers and the integration sketch given how cleanly the
protocol confirmation came out.
Thanks again for the open work, the project structure made
independent reproduction straightforward.
Nakata
Beta Was this translation helpful? Give feedback.
All reactions