Repository navigation
embed-gemma:300m returns a collapsed embedding space: deterministic, but unusable for retrieval (same model via llama.cpp is correct) #661
Description
Summary
embed-gemma:300m on the FLM backend returns deterministic but semantically collapsed embeddings. Vectors are stable across repeated requests, so this is not the instability from #647 — but the space they live in carries almost no semantic signal, which makes the endpoint unusable for retrieval.
The clearest symptom: for the query "What is the capital of France?", the sentence "Bananas are a yellow tropical fruit." scores higher than "Paris is the capital city of France."
The same model file, same server, same API layer, same client — but served through llama.cpp instead of FLM — behaves correctly. That isolates the difference to the FLM embedding path rather than to the model, the client, or the surrounding server.
FLM (embed-gemma:300m, q4) |
llama.cpp (embeddinggemma-300M-Q8_0.gguf) |
|
|---|---|---|
| cos(query, Paris…) | 0.8455 | 0.8745 |
| cos(query, Bananas…) | 0.8684 ← wins | 0.2737 |
| pairwise cos, 5 unrelated sentences (median) | 0.849 | 0.245 |
| cos(A,B) same ending, unrelated content | 0.8449 | 0.3900 |
| cos(A,C) same content, different ending | 0.9835 | 0.7598 |
| identical input → distinct vectors (10 runs) | 1 | 1 |
A pairwise cosine of 0.849 between five completely unrelated sentences is the core of it: almost everything is close to everything, so ranking degenerates to noise.
On a real retrieval benchmark (346 chunks of German technical documentation, 14 questions with a known gold chunk) FLM scored R@5 = 0.00, with the gold chunk at a median rank of 24.5 out of 50 candidates — the expected value for random ordering is 25.5. The same corpus and questions against the llama.cpp build of the same model: R@5 = 0.64.
Environment
| FLM version | v0.9.45 |
| Embedding model | embed-gemma:300m (q4, per the model docs) |
| Served through | Lemonade Server 11.5.0, flm recipe (see isolation note below) |
| Endpoint | POST /v1/embeddings |
| CPU / NPU | AMD Ryzen AI 9 HX PRO 370 (Strix Point), XDNA2 |
| OS | Debian 13 (trixie), unprivileged LXC on Proxmox, kernel 7.0.14 |
| Control group | llama.cpp (Vulkan) serving ggml-org/embeddinggemma-300M-GGUF:embeddinggemma-300M-Q8_0.gguf |
Isolation note: I drive FLM through Lemonade's flm recipe rather than bare flm serve. The control group runs through the same Lemonade instance, the same /v1/embeddings route and the same client — only the backend differs. Everything above the backend is therefore held constant. I have not reproduced against bare flm serve; if that matters for triage I am happy to.
Reproduction
stdlib-only, no dependencies:
#!/usr/bin/env python3
"""usage: python3 repro.py [BASE_URL] [MODEL]"""
import itertools, json, math, statistics, sys, urllib.request
BASE = sys.argv[1] if len(sys.argv) > 1 else "http://127.0.0.1:52625/v1"
MODEL = sys.argv[2] if len(sys.argv) > 2 else "embed-gemma:300m"
def embed(texts):
req = urllib.request.Request(
BASE.rstrip("/") + "/embeddings",
data=json.dumps({"model": MODEL, "input": texts}).encode(),
headers={"Content-Type": "application/json"})
r = json.loads(urllib.request.urlopen(req, timeout=300).read())
return [d["embedding"] for d in sorted(r["data"], key=lambda x: x["index"])]
def cos(a, b):
return (sum(x*y for x, y in zip(a, b))
/ ((math.sqrt(sum(x*x for x in a)) or 1e-9)
* (math.sqrt(sum(x*x for x in b)) or 1e-9)))
q = "What is the capital of France?"
docs = ["Paris is the capital city of France.",
"Bananas are a yellow tropical fruit.",
"The engine requires regular oil changes."]
v = embed([q] + docs)
print("1) TRIVIAL RANKING query: %s" % q)
for s, d in sorted([(cos(v[0], v[i+1]), docs[i]) for i in range(3)], reverse=True):
print(" %+.4f %s" % (s, d))
unrelated = ["The dog barks loudly in the garden.",
"Quantum physics describes subatomic particles.",
"I buy bread at the bakery every morning.",
"The stock market closed lower today.",
"Please take out the rubbish tonight."]
uv = embed(unrelated)
pairs = [cos(a, b) for a, b in itertools.combinations(uv, 2)]
print("2) ANISOTROPY median %.3f min %.3f max %.3f"
% (statistics.median(pairs), min(pairs), max(pairs)))
a = "The dog barks loudly in the garden. End."
b = "Quantum physics describes subatomic particles. End."
c = "The dog barks loudly in the garden. Beginning."
w = embed([a, b, c])
print("3) CONTENT SENSITIVITY")
print(" cos(A,B) same ending, unrelated content : %.4f" % cos(w[0], w[1]))
print(" cos(A,C) same content, different ending : %.4f" % cos(w[0], w[2]))
import hashlib
h = {}
for _ in range(10):
k = hashlib.sha1(",".join(map(str, embed([q])[0])).encode()).hexdigest()[:12]
h[k] = h.get(k, 0) + 1
print("4) DETERMINISM 10x same input -> %d distinct vector(s) %s" % (len(h), h))Actual output — FLM
1) TRIVIAL RANKING query: What is the capital of France?
+0.8684 Bananas are a yellow tropical fruit.
+0.8455 Paris is the capital city of France.
+0.7803 The engine requires regular oil changes.
2) ANISOTROPY median 0.849 min 0.798 max 0.944
3) CONTENT SENSITIVITY
cos(A,B) same ending, unrelated content : 0.8449
cos(A,C) same content, different ending : 0.9835
4) DETERMINISM 10x same input -> 1 distinct vector(s) {'9533a954a5ee': 10}
Actual output — llama.cpp, same model, same server
1) TRIVIAL RANKING query: What is the capital of France?
+0.8745 Paris is the capital city of France.
+0.2737 Bananas are a yellow tropical fruit.
+0.2008 The engine requires regular oil changes.
2) ANISOTROPY median 0.245 min 0.177 max 0.343
3) CONTENT SENSITIVITY
cos(A,B) same ending, unrelated content : 0.3900
cos(A,C) same content, different ending : 0.7598
4) DETERMINISM 10x same input -> 1 distinct vector(s) {'52707172da1b': 10}
Ruled out
Before filing I tested the obvious configuration causes; none of them explain it:
| Hypothesis | Test | Result |
|---|---|---|
| Batch reordering / index mismatch | batched vs. one-request-per-string | vectors bit-identical, cos = 1.000000 |
Missing EmbeddingGemma task prefixes (task: search result | query: / title: none | text:) |
trivial ranking with the documented prefixes | slightly worse (Paris 0.757, oil changes 0.877) |
Embedding model not co-loaded with an LLM, as the model docs require (flm serve <llm> --embed 1) |
loaded qwen3.5-0.8b-FLM onto the NPU alongside it, re-ran |
anisotropy 0.783 → 0.782, i.e. unchanged |
| A constant offset dominating the vectors | subtracted the corpus mean (centering) before ranking | median rank 41 → 61 of 120, i.e. worse, exactly random |
Non-determinism is also ruled out for this report: 20 identical requests returned 20 identical vectors on this setup.
Relation to existing issues
- Embeddings are non-deterministic: identical input returns different vectors, and processing a longer document permanently changes the embeddings of shorter ones #647 (non-deterministic embeddings, cross-request contamination) — different symptom, possibly the same root cause. That report is Windows on Ryzen AI 9 HX 370 / FLM v0.9.46; this one is Linux on Ryzen AI 9 HX PRO 370 / FLM v0.9.45, so the same Strix Point family. If the embedding path has shared state or a pooling/normalisation defect, both symptoms could fall out of it.
- embedding model fails #589 ("embedding model fails") — no detail given there; this may or may not be the same thing.
Guess at the cause
I could not narrow it further from the outside. The signature — a stable but collapsed space where content barely moves the vector — usually points at either pooling/normalisation (e.g. wrong token positions being pooled) or quantisation destroying a model too small to absorb it. The model docs list the FLM build as Q4_1, which is a coarse legacy format; a 300M encoder has little redundancy to spare, and the same weights at Q8_0 are fine. Both are consistent with what I measured, and I cannot distinguish them through the API.
Impact
This silently degrades every downstream retrieval system rather than failing loudly. In my case five RAG and agent-memory consumers had been indexing against it for months; nothing errored, results were merely subtly useless, and it only surfaced when I benchmarked retrieval quality directly. A note in the model card would already help; a fix would help more.
Happy to run further tests on this hardware if that is useful.
Activity
Atomic-Germ commentedon Aug 11, 2026
If the cause is the model being q4_1, there's zero chance of this being fixed.
Aitschend3251 commentedon Aug 17, 2026
Warning
Corrected on 17 Aug 2026. Several figures in this comment do not hold — most
importantly the "negative result" further down, which I used to rule out batch and
order dependence. It does not rule it out. The prefix claim below is wrong too, and
the length ladder cannot localise anything. See
my correction
for what is retracted and what still stands. Nothing below this line has been edited.
Independent confirmation on different hardware and a newer FLM build, plus one
observation that may help localise the cause.
Still present in FLM v0.9.46 (this report used v0.9.45), and it is not an
artefact of your container setup — my environment is bare metal, not LXC:
flmrecipeembed-gemma-300m-FLM(q4nx)llamacpprecipe,ggml-org/embeddinggemma-300M-GGUF:embeddinggemma-300M-Q8_0.ggufSame qualitative result as yours, in German instead of English: for the query
"Wie starte ich den Server neu?" ("How do I restart the server?"), the
distractor "Kaesekuchen braucht Quark." ("Cheesecake needs quark.") outscores
the correct passage.
Prompt prefixes are not the explanation. All figures here are without any
task prefix — identical on both paths, which is the clean case for isolating a
backend difference. I also ran the documented EmbeddingGemma prefixes
(
task: search result | query:for the query,title: none | text:for thedocuments) and, separately, the whole set in English rather than German. The
ranking stays wrong on FLM in every combination.
The part that might narrow it down: the collapse scales with sequence length
I ran the same six unrelated topics at four text lengths, so the only thing
that changes is how many tokens the pooling step sees. Median pairwise cosine
between the six:
Both paths drift upward with length — some of that is expected, since longer
texts share more function words. What differs is the floor and the ceiling: FLM
starts where llama.cpp ends and saturates around 0.92 from one sentence onward,
i.e. exactly where the space stops carrying usable signal. Below roughly one
sentence FLM still separates somewhat.
I offer this as an observation, not as a diagnosis — I have not looked at the
FLM source. But it is consistent with something in the pooling/aggregation step
over the sequence rather than with per-weight numerical error.
One negative result, so nobody spends time on it
I suspected the vector might depend on the other inputs in the same request
(which would have ruled out quantisation as the cause, since that is
deterministic per text). It does not. Same text alone vs. as position 1 of a
6-item batch: cos = 1.000000 on FLM. Repeating the identical single request
twice gives 0.999447 on FLM vs. 1.000000 on llama.cpp — a trace of the
non-determinism from #647, but far too small to explain the collapse.
On the "it's just q4" hypothesis from the comment above
My batch test does not rule it out. But the published artefact argues against it:
FastFlowLM/Embedding-Gemma-300M-NPU2shipsmodel.q4nxat 615.2 MB for amodel of roughly 300M parameters — about 2.0 bytes per parameter. Genuine
4-bit weights would be near 0.5 bytes per parameter, i.e. roughly 150 MB. The
file is the size of full fp16 weights despite the
.q4nxextension.I cannot read the container format, so this is an observation about file size,
not about what is inside it — the extra bytes could be scales, an unquantised
embedding table, or padding. But "the weights are too coarsely quantised to
carry semantics" is hard to square with a file that is four times larger than
4-bit weights would be.
The clean experiment is still the one that separates the two variables: an FLM
build of this model at a different precision, or the same q4nx weights through a
non-FLM runtime. From the outside, quantisation and backend change together and
cannot be told apart.
Reproduction: your
repro.pyworks unchanged against Lemonade's/api/v1/embeddings. My length-ladder variant is a few lines on top of it andI am happy to paste it if useful.
Atomic-Germ commentedon Aug 17, 2026
@Aitschend3251 If you think inspecting those container files will help: https://github.com/Atomic-Germ/FLM_Q4NX_Converter/tree/main/tools -- ironically I don't know what to look for.
Aitschend3251 commentedon Aug 17, 2026
Two things: I got several figures in my comment above wrong, and while re-checking them I found three things that are not in this thread yet. Corrections first, since one of them was a false all-clear.
What does not hold
I reported
cos = 1.000000for the same text embedded alone versus inside a batch, and used it to close the question — "so nobody spends time on it". I have since run it ten times: eight rounds give exactly 1.0, one gives 0.847, one gives 0.508. A single draw cannot tell "always" from "usually", and I had no business shutting down a line of investigation with one. That caution applies to row 1 of your own Ruled out table too, which reports the same 1.000000 — I suspect we each measured it once.Four more, briefly. My claim that all figures were taken without a task prefix is wrong — see below, FLM adds one server-side, so my FLM numbers carry it and my llama.cpp numbers do not. My length ladder cannot localise anything: same prefix asymmetry, the six texts per rung share up to 92.5 % of their words by construction, and the rungs ran in order against one server; on equal stretches both paths rise by the same amount. My cheesecake sentence is contradicted by my own table two lines below it — cheesecake came last (0.5909 against 0.6110); what beat the correct passage was the unrelated backup sentence at 0.7329. And my file-size argument was the wrong instrument: the same rule of thumb would have acquitted the Qwen3-1.7B container, which really is Q4_1. (Also, no run of mine produces the
0.999447I quoted; I cannot say where I got it. The effect behind it is real and larger than I published — two of ten repeats land at 0.94 and 0.93.)None of that touches the collapse. Six unrelated sentences still sit at a pairwise median of 0.71 against 0.24 on the llama.cpp control, and the wrong passage still wins. Happy to post the raw numbers for any of the above.
Three things that are new
1. Every input is embedded as a query, and nine of the ten task types are unreachable.
rest_handler.cpppassesembedding_task_type_t::task_queryfor every input — line 866 in v0.9.46, 891 in v1.0.1, and unchanged as far back as v0.9.36.Gemma_Embedding::embedprepends it as literal text ("<bos>" + task_prefix + text + "<eos>"), so documents are embedded withtask: search result | query:in front of them._get_task_prefixknowstask_document("title: none | text: ") and eight others; none can be reached over HTTP. That is a retrieval-quality cost independent of this bug, and it explains the prefix row in your Ruled out table: the documented prefixes you tried were added on top of the one already there.2. The batch dependency did not go away, it went quiet.
/v1/embeddingshas history here — #373 and #445 both reported that a batch returned one embedding instead of N, and the concatenation was replaced with a per-input loop in v0.9.40. What I measure now is the same family, one level down: the vector a text gets still depends on what it was sent with, only nothing looks wrong any more. Ten repeats of one request give two outliers (0.94, 0.93); alone-versus-in-a-batch drops sporadically to 0.847 and 0.508. Sharpest case: comparing a text against itself as item 0 of requests of size 1, 2 and 6 gives 0.820652 for all three sizes on v1.0.1, identical to six decimals — and size 1 is payload-identical to the first call. The vector seems to shift once and stay shifted, which is the persistence #647 describes, except there a longer document is the trigger and here two identical requests suffice.3. There look to be two defects, not one. On v1.0.1 the decision question is identical to v0.9.46 in all four decimals I record, and the pairwise median barely moves (0.7125 → 0.7173). A collapse that reproduces that exactly cannot be state carry-over — while the drops in point 2 are not reproducible at all. The llama.cpp control never leaves 0.9998–1.0 across the same 88 calls, so neither is my machine.
Two hypotheses are now off the table, for what it is worth. Quantisation: the shipped
model.q4nxdeclares 316 of 316 tensors BF16 at exactly 2.000 bytes per parameter (a 512 KB range request reads the header; the same parser finds 197 quantised tensors in the Qwen3-1.7B container, so that is a finding rather than a parser bug) — andsrc/model_list.jsonsays so itself,quantization_level: "none", the only one of 37 entries. Onlydocs/docs/models/embeddinggemma.mdline 14 printsQ4_1, which I think is where the q4 theory in this thread came from; that line is worth fixing regardless. And the prefix: applying it to every text on the control path moves llama.cpp's median from 0.241 to 0.134 — it separates better, so the forced prefix is not what creates the cone.What would help
The one test I cannot run from outside is a one-line change: swap the enum at the call site and re-run
repro.py. If the median drops, part of this is the hard-wired query prefix; if it does not, that is cleared and the search narrows to the state question in point 2. Beyond that: the docs line, and whethertask_typeshould be a request field.Happy to run anything on Linux/XDNA2 — and thanks to @Atomic-Germ, whose converter is what told me where to look in the container.
Setup: Ubuntu 26.04, Ryzen AI 9 HX 370, bare metal, through Lemonade's
flmrecipe, embedding-only (--embed 1). Your docs ask for a co-loaded LLM, but you tested that yourself — it moved anisotropy 0.783 → 0.782 — and #647 was reported with one. Version read from the running server's/api/version; the version file in the Lemonade tree turned out to be stale, which is its own small lesson. Control throughout:ggml-org/embeddinggemma-300M-GGUF:embeddinggemma-300M-Q8_0.ggufon the same server. Still not reproduced against bareflm serve.Both this comment and my earlier one were drafted by Claude Code running on my machine; I read and posted them myself. I should have said so the first time — and the mistakes above are a decent argument for saying it.
More to the point: I approved that first comment in a hurry. I did not check the numbers before it went out, and three of them turned out to have no run behind them. That is on me, not on the tool, and I am sorry for the time it cost you — particularly the "negative result", which pointed away from a question that is still open.
Atomic-Germ commentedon Aug 19, 2026
@Aitschend3251 @teconomix If you'd like to help me out with the
flm-testpython package, we can have better testingteconomix commentedon Aug 30, 2026
Thanks @Aitschend3251 — you are right on both counts that touch my report, and one of them was mine to get wrong in the first place. Corrections, then two data points from this hardware.
The Q4_1 cause in my report is wrong
I wrote that coarse quantisation was the likely cause. I took
Q4_1fromdocs/docs/models/embeddinggemma.mdand never checked it, and it is what seeded the first dead end in this thread. Verified now on my own installation, independently of yours:/opt/fastflowlm-0.9.45/model_list.json→"quantization_level": "none"forEmbedding-Gemma-300M-NPU2model.q4nxis 615,197,168 bytes for a ~308M-parameter model = 2.0 bytes/parameterThat is BF16, despite the
.q4nxextension, and it matches your 316/316 BF16 tensors from a different machine and a different FLM build.A likely origin for the docs line, which also makes it worth fixing: the v1.0.3 release notes say the Qwen3.5 family and Qwen3.6-MoE were upgraded "from Q4_1 to Q4_K". So
Q4_1was a real, current value for the LLM containers — it just does not describe this one. That reads like a copied line rather than an invented one.My "ruled out" table has one invalid row and one weak one
Row 1 (batch/order dependence) was a single measurement. Your objection stands on its own: one draw cannot separate "always" from "usually", and I presented it as settled. I have now run it properly — see below.
Row 2 (task prefixes) has the right conclusion for the wrong reason. Given your
rest_handler.cppfinding, the documented prefixes I tried were stacked on top of thetask_queryprefix FLM already prepends. The test did not measure what I claimed it measured.Two data points from v0.9.45 on Linux
Repeating your experiment here, 10 rounds each,
embed-gemma-300m-FLMthrough Lemonade 11.5.0:cos = 1.000000, no outliercos = 1.000000, no outlierSo the instability you see on v0.9.46 and v1.0.1 does not reproduce on v0.9.45 here (Ryzen AI 9 HX PRO 370, Strix Point, Debian 13 in an LXC). The collapse itself is unchanged and reproduces exactly as reported.
That is consistent with your "two defects" reading, and it bounds the second one: whatever causes the sporadic drops appears not to be present in v0.9.45, while the collapse is present in every version any of us has tested. If that holds up, the state question and the pooling question can be worked on separately.
Status here
I have since moved all consumers to the llama.cpp build and removed the FLM model from this server, so I cannot run further tests without re-pulling it — which takes about 80 seconds, so please just ask if a specific measurement would help. The one test that would settle the most is still the one none of us can run from outside: swapping the enum at the
task_querycall site and re-running the repro.And the docs line is worth fixing regardless of what the underlying defect turns out to be — it cost this thread a detour, and it cost me a wrong claim in the opening post.
Aitschend3251 commentedon Aug 30, 2026
Thanks @teconomix — your v0.9.45 rounds are one end of the range, so I measured the other: v1.0.3 (27 Aug), against a bare
flm serveinstead of through Lemonade. Neither had been tested in this thread. Numbers first, then a warning that applies to every version number any of us has quoted, mine included.v1.0.3 on Linux / XDNA2, 10 rounds per case
Ryzen AI 9 HX 370, Ubuntu 26.04, kernel 7.0.0-30, bare metal (
systemd-detect-virt→none). Server:flm serve embed-gemma:300m --embed 1, version confirmed at/api/versionon the running process →{"version":"1.0.3"}. Model pulled fresh:model.q4nxis 615,197,168 bytes, the same size you report from 0.9.45 — I compared the size only, not the contents. Control in the same run:ggml-org/embeddinggemma-300M-GGUF:embeddinggemma-300M-Q8_0.ggufvia llama.cpp.The collapse is unchanged. Six unrelated sentences, pairwise cosine:
Two of those are identical to my August runs —
min0.6502 andbest foreign0.7664, to four decimals — and the median moved 0.7125 → 0.7173 → 0.7182. The collapse reproduces exactly; it is not noise.The instability is present, and in more rounds than I have seen before:
flm serveRead that as more often, not deeper — the largest single drop is still the 0.508 on v0.9.46.
Two things this rules out. The control stayed within 0.9998–1.0 across the same run (case B: 10/10 exactly 1.000000), so it is not the machine. And case B drops with no batch involved, against a server started straight from the binary — so it is not Lemonade either, which retires the caveat I had to attach to my own numbers on 17 Aug ("still not reproduced against bare
flm serve").What it does not show: my v1.0.3 row changes two variables at once, version and serving path. The rise from 1/10 to 5/10 is therefore not attributable to the version.
A warning about version numbers, starting with mine
lemonade backends --allhere reportsflm npu installed v0.9.46while the system package is at 1.0.3. The service log says it outright:That tree's binary is dated 29 Jul. And
prefer_systemdoes not override it — I read the service's own config as root: theflmsection is{"args": "", "prefer_system": true}, and it still loads its own tree. The service runs as userlemonadewithProtectHome=yes, so this file is not the one in your home directory.So
dpkg,/opt/fastflowlm-*/, andflm --versioncan all report a version that is not answering requests. My own runs log both side by side — every one of them showsversion_txt: v0.9.46(Lemonade's tree) next to whatever the running server reported.Which is why my v0.9.46 row is marked. That run recorded no server version, only the file — so its label rests on exactly the source I am telling you not to trust. I am not withdrawing the row; I am saying it is weaker evidence than the other two, and I did not notice until I went back through the files today. If your 0.9.45-vs-0.9.46 boundary rests on a package or path version rather than on
/api/versionof the process that served the request, it is worth re-reading before we build on it — it is currently the load-bearing claim for "two defects".Two repo checks
task_queryis unchanged in v1.0.3 —src/server/rest_handler.cppline 891, same as v1.0.1 and v1.0.2 (866 in v0.9.46). The one-line experiment none of us can run from outside is not obsolete.docs/docs/models/embeddinggemma.md:14still printsQ4_1at tagv1.0.3, whilesrc/model_list.jsonat the same tag carries"quantization_level": "none". Two files in one tag, contradicting each other. Happy to send that one-line PR.For anyone reproducing against a bare server — and @Atomic-Germ, this is for
flm-testflm servefails on Linux with:flags=8209isMAP_SHARED|MAP_FIXED|MAP_LOCKED(MAP_LOCKED= 0x2000,asm-generic/mman.h), and the defaultulimit -lhere is 8 MB. It fails identically with and without--embed. It starts undersystemd-run -p LimitMEMLOCK=infinity. I cannot show that this is what stopped me in August — I did not record the error then — only that it is the wall now, and that it fails in a way that looks like the server merely not coming up.@Atomic-Germ —
flm-testrequires a running FLM server as step one, so that limit belongs in its prerequisites; on stock Linux the first test run dies before reaching any model. Beyond that, two things from this thread would fit your embedding suite, and I am happy to contribute them:Raw numbers for any of this on request.
Drafted by Claude Code on my machine; I read the numbers back against the result files and posted it myself.
Aitschend3251 commentedon Aug 30, 2026
Correction to my own comment above, and it comes from re-reading my own measurement log rather than from a new run.
I wrote that
prefer_systemdoes not override Lemonade's bundled FLM. That is what I measured today — but the belt-and-braces record from our 20 Aug run, on this same machine, says the opposite:On 20 Aug Lemonade used the system binary. Today it uses its own tree. Nothing was changed here in between — no reinstall of the backend, no edit to that config.
So the warning is not "the switch does not work". It is which binary answers can change between two measurements without anyone touching it — and it changes silently, because every other source (
dpkg, the path,flm --version) keeps reporting the system package either way.That applies to any comparison assembled across weeks, including the 0.9.45-vs-0.9.46 boundary this thread is now leaning on. The only reading that survives it is the version taken from
/api/versionof the process that served the request, at the time it served it — logged next to the numbers, not looked up afterwards.Atomic-Germ commentedon Aug 30, 2026
@Aitschend3251 - check out https://github.com/Atomic-Germ/flm-test/releases/tag/v0.6.2 ; the embedding tests work just like the others now, though you can't do both embedding and llm tests right now. I'd like any contributions to it of course!
Anyway, bleak results. But we also need to note that I'm not on the same flm as everyone else. The embedding hasn't been fiddled with in any way that would cause these though:
Important thing to note from the actual data:
Aitschend3251 commentedon Aug 31, 2026
@Atomic-Germ — thanks for wiring the embedding checks into
flm-test; E5 landingon your build is the third independent machine now, and the raw-vector CSV is the
part I did not have.
Two things about the harness, both from your own numbers rather than a run of mine.
E6 draws once, and your E7 shows what that costs
_check_cross_path_consistencytakes onesingleand onebatchembedding andcompares them against
AGREEMENT_THRESHOLD = 0.999. On a path that is stablethat is enough. On this one it is a coin flip, and the evidence is two lines
below it in the same output: your E7 run has 16 of 30 draws at exactly
1.0000. A single draw therefore passes E6 about half the time no matter how
badly the path is behaving — and it reports
cosine 1.000000, which reads likea strong result rather than one sample.
I have no standing to point this out except that I made the identical mistake in
this thread on 17 Aug: I reported
cos = 1.000000for single-versus-batch,used it to close the question, and had to withdraw it after ten rounds gave
8×1.0, one 0.847 and one 0.508. E2 failing in the same run as E6 passing is
that same shape — if repeatability is broken, agreement measured once is not a
statement about the delivery paths.
Suggestion, in the spirit of E7: draw E6
REFERENCE_DRAW_COUNTtimes as well andreport the worst and the outlier rate, the way E7 already does. It is a small
change and it would make the PASS mean what it says.
The two blocks in your comment are different runs
Your summary line says
30/30 draws deviate ... worst cosine 0.244243; the blockyou label "the actual data" says
14/30 deviatewith1.0000 ×16and a minimumof
0.1690. Those cannot be the same run: withSTABILITY_THRESHOLD = 0.999the listed values give exactly 14 outliers and a worst of 0.169000, and
0.244243 does not appear in the list at all.
That is not a counting bug — and I think it is the more interesting number in
your post. Two runs of the same harness against the same server, same input,
gave 30/30 and 14/30. The outlier rate itself is unstable between runs,
which means any single
flm-testrun is a weak basis for "this build is betteror worse than that one" — including the cross-build vector diffing the CSV is
built for. Worth putting a run count on the results file, or averaging over a
few, before anyone compares builds with it.
Not a problem: the version string
I had this on my list and it is wrong, so for the record:
_get_flm_version()reads
{base_url}/versionfrom the running server, not the CLI. That is exactlythe source I argued for above, so results files carry the version that actually
served the request. No change needed.
Happy to send a PR for the E6 change if you would rather have code than a
comment — say the word and I will keep it to that one function.
Atomic-Germ commentedon Aug 31, 2026
@Aitschend3251 I certainly wouldn't mind a PR! And you're right those numbers are from a separate run -- but the better of the two was with some fixes, specifically to see if results improved. That same degradation seems to happen regardless, currently.
ALSO -- I think I might have a working gemma embedding xclbin (well a little more than that). At least I'm pretty sure. Testing.
Edit: Promising results.
embedding_results_v1.0.1-add1.csv
EDIT AGAIN: I've got a helpful opencode skill now: https://github.com/Atomic-Germ/FastFlowLM_v1/blob/embedding/.opencode/skill/npu_offload_pipeline.md
Aitschend3251 commentedon Sep 2, 2026
@Atomic-Germ — the part of this worth naming isn't the green suite, it's what sits under it.
gemma3_reference.pycarries the model card's own anchor —[0.3011, 0.6359, 0.4930, 0.4889]— and checks against it to ±0.005, and your engine then agrees with that oracle to worst-cosine 0.999993 over eight texts. Nothing else in this thread is pinned to a published number at both ends.That also fixes what it does and doesn't say, and it's worth being explicit because the numbers are good enough to be over-read. Per
make_manifest.pythe engine loads the originalgoogle/embeddinggemma-300msafetensors straight out of the HF cache, so both sides of E8 read the same upstream weights and neither side touchesmodel.q4nxor the shipped kernels. The green suite is a statement about your new path, not about the one this issue was filed against — which is a good result, just a different one. It is the cleanest evidence yet that the weights and the silicon are fine.Two things I can't tell from outside, so as questions rather than findings:
How much of it ran on the NPU? From
engine.cpp:45I readq_proj,o_proj,gate,upanddownoffloaded, with k/v and the Dense head kept on CPU on purpose — the comment above it says for E8 fidelity — and the scores/mask/softmax/AV core still fp32 CPU between:399and theo_projcall at:455. Your own server answers this in one line: does that run printNPU matmul ready (N bf16 weights)orrunning CPU-only(engine.cpp:116/:135)? Related,1.0.1-add1puts the build before your 19:51 bump to-add2, but that still straddles the 19:29 offload commit.Can
matmul_bf16fail quietly?matmul_t_npufalls back tomatmul_twithout a word (engine.cpp:243-273). An intermittently failing dispatch would look like "mostly identical, occasionally not" with nothing in the log — probably worth a line on stderr whatever else is going on.On which: one line in your skill may be the first mechanical explanation anyone has offered for the other defect —
— not for @teconomix's report here, which ruled non-determinism out explicitly, but for #647 and for my own rounds (8 of 10 draws at exactly 1.0, then 0.847 and 0.508). It isn't the only candidate: #647 ties five of six corrupt responses to 9–39 s latency spikes and reports
--pmode turbocutting 8 distinct vectors to 2, neither of which a missing output sync would do. Cheap for a maintainer to rule in or out, though, and it's more than anyone had before.E6 PR is coming — you already said yes, so I'll keep it to that one function.
Drafted with Claude Code. The measurements and the code references are my own, re-checked before posting.
abn commentedon Sep 5, 2026
Same collapse on a fourth machine, and it survives v1.0.4. Two things I have not seen in the thread: the v1.0.4 engine is the same binary as v1.0.3, and the repo already ships a reference vector that the Linux build does not reproduce.
xrt-smi: RyzenAI-npu4, aie2p, 6x8ulimit -lunlimitedflm serve --embed 1; v1.0.4 bare, from the Linux tarball; v1.0.4 HRX build (flm-hrx). Versions read from/api/versionon the serving processEmbedding-Gemma-300M-NPU2,model.q4nxsha256 matches the HF uploadunsloth/embeddinggemma-300m(ungated mirror ofgoogle/embeddinggemma-300m) through Sentence Transformers 6.0.1, CPUv1.0.4.
lib/libgemma_embedding.soand the four xclbins forEmbedding-Gemma-300M-NPU2in the v1.0.4 Linux tarball have the same md5 as in v1.0.3. Output on v1.0.4: "What is the capital of France?" scores 0.850 against the Paris sentence and 0.873 against the bananas sentence. Same ordering as the opening post.The reference vector.
src/test/gemma_embedding/test.cpphas a 768-float array for the Alice paragraph, embedded astask_query. Against the Sentence Transformers reference it scores cosine 0.9977 with the full pipeline (mean pool, both Dense layers, normalize) and -0.015 with the Dense layers removed. So the engine did produce the right vector on whichever machine that array was captured on. The same text through the live v1.0.3 server here gives 0.003 against that array, three draws out of three. I could not runtest_gemma_embeddingitself: built from July source against the current libraries it dies withstd::bad_allocinload_weights, same as the repo-built v0.9.43flm. If someone at ROCm can run it on a Linux box, its error metrics against that array answer most of this thread.On the 316 tensors: 314 model tensors plus
dense_2.weight[3072,768] anddense_3.weight[768,3072], andlibgemma_embedding.socontains both names. This matters for the control group as well. llama.cpp GGUFs converted before April 2026 have no Dense tensors and land in a different space, cosine about 0 to the real model, so a comparison against one of those measures the wrong thing. The ggml-org Q8_0 re-uploaded on 2026-04-29 has them.What the output is not. I swept the reference model for anything the engine could plausibly be returning: token embeddings only, the hidden state after each of the 24 layers (mean pooled, Dense, normalized), the final state without Dense, first-token and last-token pooling, all of it with and without the
task: search result | query:prefix. Best cosine to the FLM output anywhere in that sweep: 0.12. Not a stopped-early layer, not pooling, not the missing Dense head. And not the prefix: the reference side carried the same prefix the server prepends and still matched nothing. Thetask_queryenum atrest_handler.cpp:891is still wrong for documents, it is just not what collapses the space.It is not the XRT runtime layer either. The same test through the HRX build (
flm-hrxv1.0.4, linked againstlibhrx.so.0, no XRT libraries mapped, its own build oflibgemma_embedding.so) gives 0.003 against the reference array, three draws out of three, and 0.847 for Paris against 0.874 for bananas. Both packages here load the same four xclbins, same md5. Two runtimes, two engine builds, identical kernels, identical wrong output, so what is left is the kernels themselves or the host-side embedding logic they share.Two more bounds.
flm serve qwen3:0.6b --embed 1answers chat prompts correctly in the same process while the embedding output stays the same to four decimals, and the amdxdna log is clean during the calls. Over-length input is truncated, not rejected:modeling_gemma_embedding.cppcuts to 2047 tokens plus<eos>, and a 6.6k-token input comes back 200 withusage: 0.Vectors here are unit norm (0.997), deterministic across runs, pairwise 0.64 to 0.92 between unrelated sentences. Raw vectors and the sweep script on request.