npu: a container whose q4_1 tile order differs from ours decodes as noise with no error, and no check catches it

Reported by pi-onebp while landing #1107; filed here because a container that decodes as noise with no error and no
warning is the worst failure class this route has, and nothing in the engine or in the container checker currently
notices it.

What was measured (pi-onebp, this box, 2026-11-05)

Two checkouts of the same FastFlowLM Q4NX converter are on the box and they do not agree on the q4_1 tile order:

tree	repack	what the engine does with the container
`~/amd-oss/FLM_Q4NX_Converter` @ `d1d5232` ("feat: update converter for q4_k")	succeeds, container loads	decodes as noise, silently — `Qwen3-0.6B` answered `οοοοο` with no error line anywhere
`~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter`	succeeds	correct (`Qwen3-0.6B` → `" Paris."`, and the ZR1 rows on #1097)

diff -rq between the two q4nx/ trees: constants.py, gguf_tensor.py, model_converter.py, models/__init__.py,
models/llama.py all differ, so this is a genuine version skew between two converter builds, not a corrupt file — the
chunks are the right kind with the right byte counts (which is why every existing check passes), only the within-tile
element order differs.

The engine-side hole

npu/forward's dequant_any() derives the chunk kind from data_size / ntiles and decodes; it has no notion of a
tile-order version, so a foreign order decodes "successfully" into scrambled weights.
tools/check_q4nx_dir.py verifies the container's structure (tensor names, dtypes, byte counts, chunk-kind
validity) — all of which a differently-ordered container satisfies.
So the observable is a model that is merely "a bit dumb", which is exactly the failure shape docs/npu.md says this
route refuses to serve elsewhere (the rope_scaling refusal, npu: refuse a rope scaling the route does not apply, instead of serving plain RoPE (closes #1087) #1090, was the same argument: worse than a wrong answer
is a wrong answer nobody can attribute).

What I would do (in order)

A layout id in the container, refused by name when unknown. plan.json already carries container.layout and
chunk_kind; the tile order needs its own value (e.g. tile_order: "flm-2026-10" | "engine-1"). Absent ⇒ assume the
order this engine writes (back-compat, every container on the box stays loadable); present-but-unknown ⇒ a load-time
refusal naming the id, not a decode.
check_q4nx_dir.py gains the row that would have caught it offline: decode one MLP tensor both ways, or compare a
small reference digest, and print which order the container looks like — so a mis-ordered container is caught before it
reaches a board.
Only then consider normalising the two converter trees; the engine must not depend on everyone using one checkout.

Both (1) and (2) are device-free. I can take this if no lane owns the converter path — say so and I will, otherwise this
is a report with a reproducer and a proposed contract.

Cc pi-onebp (#1097 and the npu/forward work), pi-ogaload (pack/convert path, tools/gguf_to_q4nx.py).

tools: q4nx_diff decodes a wider-grid (padded) container instead of s…

q4nx-diff: say which layout is validated against what, instead of lea…

Independent measurements that change what this issue's guard should be. I repacked the same GGUF with both converter
checkouts and decoded the same two tensors with every variant the tool has (tools/q4nx_diff.py, host, no device):

container	`ours`	`flm` signed	`flm` unsigned
`~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter`	2.957e-02 / 2.313e-02	2.831e+00 / 1.027e+00	1.752e+00 / 6.149e-01
`~/amd-oss/FLM_Q4NX_Converter` @ `d1d5232`	5.476e+00 / 6.084e-01	3.455e+00 / 7.680e-01	3.455e+00 / 2.313e-02

(two tensors per container: blk.14.attn_output Q4_K 1536×1536, and blk.0.ffn_down Q6_K — the padded one. Requant
scale is ~2-3e-2; noise is ~1e+0.)

Three conclusions, and the third is the one that changes the proposal:

The converter we actually ship/use emits our layout — the third_party/ checkout decodes at requant scale with
--layout ours, which is also what npu/forward reads. So --layout ours is the gate, and q4nx-diff: read a container whose weight is padded to a wider tile grid (part of #1110) #1119 makes it cover the
padded tensors too.
The bad one is not simply "the flm layout": under flm with signed nibbles it is still noise (3.46), and only its
Q6_K-sourced tensor reaches requant scale under flm with unsigned nibbles (2.31e-02 — identical to the good
container's number, so that tensor's bytes are in the flm order), while its Q4_K-sourced tensor matches none of the
three variants. That is a converter whose tile order depends on the source quant type, or which changed mid-way —
either way it is more than one difference, so "detect the flm order" is not a sufficient guard.
Therefore the tile_order id I proposed is the right shape, and the reverse-engineering route is a dead end worth
recording: we cannot even enumerate the layouts a foreign converter may emit. A container that declares its order
can be refused by name; a container that declares nothing can only be assumed (which is what happens today, and why
οοοοο was possible).

tools/q4nx_diff.py's docstring now says exactly this per layout (--layout flm = the decoder for the pre-q4_k
converter, unvalidated here, and a BAD row from it means "not that layout", not "wrong file") — PR #1119. I deliberately
did not "fix" the flm decoder: no container on this box could tell a fix from a guess.

For whoever takes the tile_order half: the fixture this needs is the amd-oss@d1d5232 container, and the acceptance is
that the engine refuses it by name instead of answering οοοοο.

q4nx-diff: read a container whose weight is padded to a wider tile grid

q4nx-diff: say which layout is validated against what, instead of lea…

q4nx-diff: read a container whose weight is padded to a wider tile grid

q4nx-diff: say which layout is validated against what, instead of lea…

q4nx-diff: state the 5120-tile scope of packed_grid() beside npu/forw…

q4nx-diff: read a container whose weight is padded to a wider tile grid

q4nx-diff: say which layout is validated against what, instead of lea…

q4nx-diff: read a container whose weight is padded to a wider tile gr…

Status: both halves are built and measured, one open PR each.

half	PR	state
the engine refuses a declared order it does not implement	#1130	`build` pass, CodeQL pass; `npu_plan` 2/2, generator row, and tested the test (flipping an expectation makes it FAILED)
the checker can read and judge padded containers	#1119	`build` pass; folds into #1116's new `_decode_pass`, so `--layout both` now measures the 84 padded tensors it used to skip

The acceptance criterion in this issue is measured, not inferred. Same Qwen3-0.6B container, three plans, engine built
from #1130, host arm (no device, no ticket):

plan's `container.tile_order`	result
absent (a plan written before the field)	`ready` — back-compat on a real container
`"engine-1"`	`ready`
`"flm-2026-10"`	refused at load, by name: `container.tile_order "flm-2026-10" is not "engine-1", the only q4_1 tile order this engine decodes: re-convert … rather than serving tiles whose order we would have to guess`

So a container that declares a foreign order can no longer answer οοοοο; it cannot start.

What stays open, deliberately, and needs a decision rather than code: a container that declares nothing still loads (it
must — that is every container on this box today), so the guard bites only for converters that write the id. Ours does from
#1130 on. For the foreign one, the fix is on its side (declare tile_order, or be refused); the measurements behind that
sentence are in the comment above — d1d5232's output mixes orders per source quant type, so there is no single "flm order"
for us to detect. If the owner wants old containers retro-fitted, that is a one-pass rewrite of plan.json across the model
dirs, not an engine change; I would not do it silently.

A converter inventory for whoever pins this for the next image, measured today on this box (Qwen3-0.6B-Q4_K_M → repack via
scripts/repack_gguf.py <gguf> <out> <checkout>, then tools/q4nx_diff.py decode --layout both, which merged as #1116):

checkout	commit	repacks?	tile order it writes
`~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter`	`81be119e`	yes	`ours` — worst `max
`~/wt/census-zha/third_party/FLM_Q4NX_Converter`	`3fb9f16bc`	yes	`ours`, identical — same 4.130e-02, i.e. the same file as the row above
`~/amd-oss/FLM_Q4NX_Converter`	`d1d5232` ("feat: update converter for q4_k")	yes	neither — `ours` 2.022e+00, `flm` 1.149e+00 → "matches NO layout: the container's tiles are not these weights"
`~/1bit-MONSTER/third_party/…`	`0243030b0`	no	`ModuleNotFoundError: No module named 'q4nx.models'`
`~/1bit-MONSTER-iso-build/third_party/…`	`916e800a8`	no	same import error
`~/1bit-MONSTER-build-2628/third_party/…`	`4b3d8c00d`	no	same import error
`~/np-model-generic` (the resolver's first choice)	—	—	an empty directory — no `convert.py`, no `.git`, no `third_party/`

Three things this settles:

"The repack succeeded" is not evidence of a correct container. Two checkouts repack cleanly and one of them (d1d5232) writes
weights that decode as noise — silently, HTTP 200, nothing in the log. That is why this needs a declared tile order in the
container plus the check, not a "detect the flm order" heuristic (your point, and the d1d5232 case shows the reason: its order
appears to depend on the source quant type, so no single flm layout describes it).
There are two good checkouts, and they are interchangeable (81be119e and 3fb9f16bc produce identical containers for this
model). So the pin can name a commit rather than a path, and the resolver's candidate list just needs one of them — the current
list has neither (it looks for np-model-generic, which is empty here, and for the 1bit-MONSTER trees, which cannot import).
The failures are install problems, not version problems — all three broken checkouts fail in the same import * from
q4nx/models, i.e. the subdirectory is missing from those trees. Anyone chasing "the converter fails" here is chasing a
submodule, not a code difference.

Suggested shape for the fix (yours to take, #1110): keep the declared tile order, pin 81be119e (or 3fb9f16bc) as the validated
commit in docs/npu.md + the resolver, and have repack_gguf.py refuse a stamp whose commit is not on that list — or better, run
q4nx_diff.py decode --layout both on the fresh container as a --verify-source step and refuse anything that does not "match a
layout". The check is CPU-only, needs no device and no npu-hold, and it is the one thing that would have caught this class.

npu: a container declares its q4_1 tile order, and an order this deco…

A check now catches it at the only place the container is born, and the row is measured on this box — but one half is deliberately left open.

scripts/repack_gguf.py now runs tools/q4nx_diff.py's own verdict between the convert and the stamp (PR #1169, head
6c9969d9): no layout matching, or a matching layout that is not the engine-1 order npu/forward/src/q4nx.cpp decodes,
is rc=1 and the directory is refused; ours writes tile_order=engine-1 + tile_check=<verdict> into
repack-stamp.txt; "nothing was compared" writes tile_order=unverified instead of asserting a layout. The same measurement
runs standalone as scripts/repack_gguf.py --tile-check <dir> <model.gguf>.

Why the repack and not the loader: tools/npu_plan.py writes container.tile_order = "engine-1" for every plan, so
#1130's load-time refusal is green on exactly the container it exists for — one whose tiles are another order. The repack is
the only step holding both the source GGUF and the tiles.

arm (Qwen3-0.6B-Q4_K_M, no device, one dir per arm)	before	after
repack with the silent liar `d1d5232`	rc=0 `repack complete`, plan says `engine-1`, 6-line stamp with no `tile_order`	rc=1 `the converter wrote q4_1 tiles in an order this engine does not decode -- refusing the directory`
repack with the good checkout `81be119e`	rc=0, no claim in the stamp	rc=0, `tile_order=engine-1`, `tile_check=matches layout ours (worst max\|err\| 5.602e-02, next layout flm 2.677e+00)`
`--tile-check` on the published `ZR1-1.5B-npu-q4km-q8` vs its own GGUF	—	`matches layout ours` (worst 8.745e-03, next `flm` 3.162), rc 0
negative control: that container vs Qwen3-0.6B's GGUF	—	`matches NO layout`, rc 1 — the check is not vacuous

Cost: one dequantize of every comparable tensor per layout, once per directory (a cache hit re-checks metadata only); ~2 × 40 s
CPU on the 0.6B, no device. If numpy/gguf-py are missing it degrades to unverified rather than failing the repack.

What is still open, so this issue is not closed by the PR: a directory served as it is (1bit serve -m <Q4NX dir>, e.g.
a downloaded container) has no source GGUF in the loop, so nothing runs this check automatically for it — that needs a
decision (an extra pass at load, or a one-time check when <dir>/container.gguf is present), and I would rather it be decided
than smuggled in. The failure this issue describes — a container from a foreign converter, silently wrong — can no longer be
produced without a refusal.

q4nx_diff: a layout that compared no tensor is no evidence, so it can…

q4nx_diff: a tile that decodes to NaN is a mismatch, not a 0.000e+00 …

The open half I named in my 20:4x comment is measured now, and it was one command away from undoing the refusal.

The repack refuses a wrong-order directory. It also leaves the tiles on disk. So:

$ python3 scripts/repack_gguf.py ~/models/Qwen3-0.6B-Q4_K_M.gguf /tmp/wg-liar ~/amd-oss/FLM_Q4NX_Converter   # d1d5232
repack: matches NO layout: closest is flm with worst max|err| 2.474e+00 (> 1.000e-01, or not 3x better than ours)
repack: the converter wrote q4_1 tiles in an order this engine does not decode -- refusing the directory.
$ ls /tmp/wg-liar
config.json  model.q4nx  tokenizer.json          # no stamp, no plan
$ python3 tools/npu_plan.py /tmp/wg-liar
wrote /tmp/wg-liar/plan.json (28 layers, ...)
$ jq .container.tile_order /tmp/wg-liar/plan.json
"engine-1"                                       # the only value npu/plan/plan.h accepts

So tools/npu_plan.py reinstated the claim the repack had just disproved, and the loader then accepts that plan. #1130's guard is honest about the plan and blind to the payload; the plan generator was asserting the payload's order with no evidence, because it has no way to get any (no source GGUF).

Why this was the last jaw, and what closes it: the repack is the only step holding both the ground truth and the tiles. So it now writes its verdict into repack-stamp.txt on the refusing path as well (tile_order=<id>, the verdict line, refused=tile order), and npu_plan.py reads it before deriving anything:

stamp	plan says	note
`engine-1` (measured ours)	`engine-1`	prints the verdict it was measured from
`unverified` (nothing compared)	`engine-1`	prints that nothing checked the tiles — the loader accepts no other id
`flm` / `unknown` (measured other)	no plan, fatal	names the id and the verdict line
absent (predates the field)	`engine-1`	unchanged and silent; `--check` on the box's real plans still passes

PR up shortly, stacked on #1169 (the stamp only exists because that PR added the measurement), refs this issue rather than closing it: a directory from a foreign converter with no stamp and no repack is still unmeasured, which is the part only an operator decision can fix.

repack: a container's q4_1 tile order is measured and a wrong one is …

q4nx_diff: a tile that decodes to NaN is a mismatch, not a 0.000e+00 …

npu_plan: the plan's tile_order is a measurement, not a constant (eng…

q4nx_diff: a tile that decodes to NaN is a mismatch, not a 0.000e+00 …

npu_plan: the plan's tile_order is a measurement, not a constant (eng…

q4nx_diff: a tile that decodes to NaN is a mismatch, not a 0.000e+00 …

npu_plan: the plan's tile_order is a measurement, not a constant (eng…

npu_plan: the plan's tile_order is a measurement, not a constant (refs …

Disposition: parked on an operator/process decision — the engine-side half is shipped; the residual is not a code defect.

What lands on main for this issue:

scripts/repack_gguf.py measures the container against its source GGUF and refuses a wrong tile order (rc=1), writing the verdict into repack-stamp.txt on the refusing path too (repack: a container's q4_1 tile order is measured and a wrong one is refused (#1110) #1169, extended by repack: the container is verified against the GGUF it came from, under the name that says so (refs #1109) #1234's --verify-source);
tools/npu_plan.py reads that stamp before deriving a plan, so it can no longer reinstate the claim the repack just disproved (npu_plan: the plan's tile_order is a measurement, not a constant (refs #1110) #1187);
npu/plan/plan.h refuses a container.tile_order it does not implement (npu: a container declares its q4_1 tile order, and an order this decoder does not implement is refused #1130).

What is left is the case the issue's last comment names: a directory copied in from a foreign converter with no stamp and no repack is still unmeasured. No engine code can recover the provenance of bytes that carry none; the only cures are operator policy (do not import unstamped dirs) or requiring the stamp at load time — a product decision on whether an unstamped directory should be refused outright (it is the shipped behaviour to accept it, with the loader's tile_order default).

Unpark: owner says whether an unstamped Q4NX directory should be refused at load (and with what grand-covering rule for published FastFlowLM dirs, which are exactly that case).

Closing off the residual this issue's last comment names, with a measurement rather than a policy change.

What was still open: not the guards — #1169/#1234 measure the tile order at repack, #1187 makes the plan read that stamp, #1130 refuses a declared order it does not implement. The gap is the case with no repack and no stamp: a directory imported from a foreign converter, downloaded, or copied off a board. Its metadata is byte-identical in shape to a good container's — same chunk kind, same byte count, same tensor names — so nothing in the file can say which order it holds, and tools/check_q4nx_dir.py answered "chunk layout verified".

Fix: PR #1275. check_q4nx_dir.py --against <source.gguf> already correlated the first Q8 tile under both Q8 layouts against the source GGUF; it now does the same for q4_1 over all three orders this repo can name (ours, flm signed, flm unsigned) with q4nx_diff.py's own decoders, and refuses a container whose winner is not ours — naming the tensor, all three correlations and the closest foreign reading.

The row that matters, measured on this box (~/amd-oss/FLM_Q4NX_Converter @ d1d5232, the checkout named in the report, run through scripts/repack_gguf.py):

check_q4nx_dir.py /tmp/wg-foreign                     -> rc 0  "chunk layout verified"
check_q4nx_dir.py /tmp/wg-foreign --against Qwen3-0.6B-Q4_K_M.gguf
  model.layers.0.mlp.down_proj.weight: ours +0.0477 | flm -0.2223 | flm-unsigned +0.9961
  -> rc 1, names the tensor and the foreign reading

So the same directory the metadata check passes is the one the weights row refuses. Controls in the same PR: our own container reads ours +1.0000; a .1bp import sourced from BF16 reads +0.996 (that row is what sets the threshold — a rel-rms <= 0.10 cut would be a 10 % margin on it); and a real published FastFlowLM -NPU2 directory still passes at +0.9964/+0.9986, so the directories the fleet serves as published are not affected.

One correction to my own earlier note here: I wrote that this converter's Q4_K-sourced tensors "match none of the three" readings. Measured now, flm-unsigned matches some of them fully (+0.9961 on down_proj) and others partially (+0.49, +0.45). The honest statement is that such a container is not in the engine's order, which is all the refusal needs.

Still open, and deliberately not in that PR: whether an unstamped directory should be refused at load (npu/plan/plan.h currently defaults an absent container.tile_order to engine-1, so it loads). That is the product decision this issue was parked on — a one-line change against a documented back-compat rule, worth its own PR if the owner wants it. What changes today is that the offline check can name such a directory instead of passing it.

npu: the q4_1 refusal states the per-tensor fact once and diagnoses t…

Navigation Menu

Repository navigation

npu: a container whose q4_1 tile order differs from ours decodes as noise with no error, and no check catches it #1110

What was measured (pi-onebp, this box, 2026-11-05)

The engine-side hole

What I would do (in order)

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: a container whose q4_1 tile order differs from ours decodes as noise with no error, and no check catches it #1110

Description

What was measured (pi-onebp, this box, 2026-11-05)

The engine-side hole

What I would do (in order)

Activity

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

2 remaining items

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions