npu: a GGUF repacked by the GGUF-on-NPU path is a container the per-op forward cannot consume (every request: 'forward step failed: dequant failed')

`serve -m <gguf> --device npu` repacks a Qwen2.5 GGUF into a container the per-op forward cannot consume: every request fails with `forward step failed: dequant failed`

docs/npu.md (~line 499) says GGUFs of six architectures, Qwen2.5 among them, "answer correctly this way" on the generic
per-op path. This one does not. Model: Zyphra/ZR1-1.5B (MIT), model_type: qwen2, 28 layers, H=1536 IM=8960 NH=12 NKV=2
HD=128 V=151936, attention_bias: true, tie_word_embeddings: false.

Binary da845311… (origin/main bd7d451a), driver 646FFC30… and 349FDC99… (both tested), every arm in its own
npu-hold ticket, bare serve, SIGTERM+wait.

Reproduction

ONEBIT_Q4NX_CONVERTER=$HOME/amd-oss/FLM_Q4NX_Converter \
ONEBIT_NPU_KERNELS=$HOME/models/npu/zr1-elfs \
1bit serve -m ZR1-1.5B-Q4_K_M.gguf --device npu -p 18182

repack: FLM converter, configs/qwen2.json, all 28 layers mapped → ~/.cache/1bit/q4nx/ZR1-1.5B-Q4_K_M
(1.45 GB, 339 tensors, no plan.json)
load: fine. weight layout: rm, dims from config, derived 4 designs, all 8 kernels full ELF ok,
decode family: 4 role(s) at M=16, separate lm_head.weight -> dequantized (233373696 floats), ready
every request fails in 6–9 ms, HTTP 500, both prompt shapes (plain string, and token ids per forward-serve: /v1/completions takes the prompt as token ids, so a route parity run measures the forward #938):
{"error":{"message":"forward step failed: dequant failed"}}

Isolation: it is the container, not the route, the kernels or the device

Serving that same model.q4nx with forward-serve directly (no serve wrapper), two arms in one ticket:

arm	result
device GEMM (the 8 ZR1 ELFs)	500 `forward step failed: dequant failed`, 8.8 ms
host GEMM (`NPU_INFER_HOST_GEMM=1`, no dispatch at all)	500 `forward step failed: dequant failed`, 6.2 ms

Control, same binary, same driver, same request: this lane's own containers built from the same GGUF by
tools/gguf_to_q4nx.py answer — ZR1-1.5B-npu-q4km-v2 (197/197 q4_1) returned 200 with 8 tokens on 349FDC99 minutes
earlier, and ZR1-1.5B-npu-q4km-q8 (168 q4_1 + 29 Q8) has an 8-prompt route row recorded on #799. So the model, the ELFs,
the geometry, the request shape and the device path are all fine; the repacked container is what fails, and it fails before
any dispatch happens.

What I ruled out, so nobody re-runs it

tools/q4nx_diff.py compare and tools/check_q4nx_dir.py on both containers:

property	published (`gguf_to_q4nx.py`)	repacked (FLM converter)
tensor names	339	identical set (only-A 0, only-B 0)
dtype per tensor	I8 quantized / BF16 rest	identical
per-tensor byte count	—	identical (e.g. `q_proj` 1 474 560 B both)
chunk kinds	168 q4_1 + 29 Q8	197 q4_1
`check_q4nx_dir.py`	verified	verified ("every quantized tensor's shape names a known chunk kind")
tile shape	3-D `[48, 6, 5120]`	2-D `[288, 5120]`
`__metadata__`	present	absent
`config.json`	`q4nx_weight_layout: "rm"`, `q4nx_source`	neither; all dims and `attention_bias` identical
`plan.json`	present	absent (probes decide, and they decide correctly: `fused_gu`, `family rm`, M=16 decode)

2-D shape is not the cause. dequant_any() (npu/forward/src/q4nx_forward.cpp:180) derives the chunk size from
data_size by design ("so a 2-D [ntiles, N] and a 3-D [ntr, ntc, N] tensor both work"), and the forward never reads
a tensor's shape at all (grep -n 'ndim\|shape\[' npu/forward/src/q4nx_forward.cpp → no hits).
All-q4_1 is not the cause. ZR1-1.5B-npu-q4km-v2 is 197/197 q4_1 and serves.
K and N are tile-aligned for every ZR1 projection (H=1536, IM=8960, V=151936), so a padding mismatch is unlikely too.

The defect that outlives the root cause

load_layer() returning false sets the bare string dequant failed (q4nx_forward.cpp:4710, :3700), so neither the log
nor the HTTP body names which tensor, which data_offsets, which data_size failed — the one piece of information that
would bisect this in a minute. Naming it (or logging it where dequant_any returns non-zero) is worth doing regardless of
what this container turns out to be; it is also why this report can say "the repacked container" and not "tensor X".

Ask

Whoever owns npu/forward/the repack path: both containers are on this box (~/.cache/1bit/q4nx/ZR1-1.5B-Q4_K_M and
~/models/npu/ZR1-1.5B-npu-q4km-v2), the failure is deterministic under forward-serve in ~5 s, so a --kernels-free host
arm plus a print in load_layer should land it. Remaining suspects, in order: the absent __metadata__, the repack's
config.json (no q4nx_weight_layout — the probe defaults to rm, which is what the log reports), or the attention_bias
tensor set, which I believe no other model on this box has.

Repeatability, since the first report: the failure is deterministic across four tickets and two drivers, always under
forward-serve on that container, always the same message, and the two arms (host vs device) are independent servers:

ticket	driver	device arm	host arm
06:18 / 06:21	`646FFC30…`	500 `dequant failed`	500 `dequant failed`
06:35:03	`349FDC99…` (0011 v2)	500 `dequant failed`, 21.8 ms	500 `dequant failed`, 7.1 ms
06:37:32	`349FDC99…`	500 `dequant failed`, 8.8 ms	500 `dequant failed`, 6.2 ms

In the same tickets the control container (~/models/npu/ZR1-1.5B-npu-q4km-v2, packed from the same GGUF by
tools/gguf_to_q4nx.py) answered 200 with 8 tokens on 349FDC99 — so the comparison is same-binary, same-driver, same request,
same ELF set. Nothing here depends on the driver patch series or on 0011/0011v2.

One extra data point that narrows the suspects: the repacked config.json is 425 B and carries rms_norm_eps: 9.999999974752427e-07
(the raw float32 of 1e-6) while the published container's says 1e-06. Same value, so not it — but if the forward parses eps or any
other numeric with a fixed-precision reader, this is the one field whose textual form differs between the two containers.

Root cause found (arithmetic, no code change needed to see it): the FLM repack pads `intermediate_size` to the 512-column grid, the forward pads it to the 256 grid, and `dequant_any()` then computes a chunk size that is not a chunk size

I audited both containers' headers against the tile geometry the forward assumes — Q4NX_TILE_ROWS 32, Q4NX_TILE_COLS 256,
Q4NX_TILE_BYTES 5120 (npu/forward/include/q4nx_dequant.h:49-53), and q4nx_pad_k()'s 256 grid vs q4nx_pad_n()'s
512 grid (npu/forward/src/q4nx_pack.c:115-121) — for every I8 tensor, deriving the expected rows × cols from the model dims
(H=1536, NH·HD=1536, NKV·HD=256, IM=8960, V=151936) and checking n = (rows/32)·(cols/256), cb = data_size/n:

container	I8 tensors	anomalies
published `ZR1-1.5B-npu-q4km-v2` (serves)	197	0
FLM repack `~/.cache/1bit/q4nx/ZR1-1.5B-Q4_K_M` (fails)	197	84 = 28 layers × 3 MLP tensors

Every anomaly is the same shape, and it is only the MLP tensors (all attention tensors are tile-aligned either way):

model.layers.0.mlp.gate_proj.weight   shape [1728, 5120]  data_size 8 847 360
                                      expected 280×6 = 1680 tiles (IM 8960 × H 1536)
                                      container         1728 tiles  ⇒ 288×6 : IM padded to 9216
model.layers.0.mlp.down_proj.weight   shape [1728, 5120]  expected 48×35 = 1680 ⇒ 48×36 : IM padded to 9216

8960 = 17.5 × 512, and 9216 = 18 × 512: the FLM converter rounds intermediate_size up to its 512-column grid on
both sides (gate/up's N and down's K). The forward, on the other hand, pads K to 256 (q4nx_pad_k(8960) == 8960) and then
reads the tensor with the unpadded geometry — deqT(&lw->down_proj_weight, wd_.data(), H, impad_) with impad_ = 8960
(q4nx_forward.cpp:581, :1955). So dequant_any() does

n  = (1536/32) × (8960/256) = 1680
cb = data_size / n          = 8 847 360 / 1680 = 5 266  (truncated)
5266 ∉ {5120, 4736, 8704}    ⇒ falls through to the legacy non-chunked q4nx_dequant_tensor()
                             ⇒ load_layer() false ⇒ step(): "dequant failed"

which is exactly the 6–9 ms failure, and it explains why the host-GEMM arm fails identically (the dequant is host-side).

Why this is bigger than ZR1

The class is any model whose intermediate_size is not a multiple of 512 — which includes the Qwen2.5 sizes the doc names
(docs/npu.md:499: 1.5B IM=8960, 3B IM=11008, 0.5B IM=4864 — all x.5 × 512), so "GGUFs of six architectures answer correctly
this way" holds only for models that happen to be aligned (7B's 18944 = 37×512). ZR1-1.5B is a Qwen2.5-1.5B finetune and
inherits exactly that geometry.

Smallest fix, in order of preference

dequant_any() derives the tile count from data_size, not from the geometric grid: if nbytes % 5120 == 0 take
cb = 5120, n = nbytes/5120 (same for 4736/8704) before falling back to the geometric n. That turns "chunk size
not recognised" into a correct decode for a padded grid, and is ~5 lines in one function; the geometric path stays for
everything already working (I verified the arithmetic reproduces 5120 for both containers: 1 680 and 1 728 tiles).
Make the forward's impad_/N-padding use q4nx_pad_n()'s 512 grid for the MLP tensors so it matches the converter, and
keep a fallback when the container is unpadded.
Make the repack stop padding (scripts/repack_gguf.py/the converter) so the container matches gguf_to_q4nx.py's layout —
cheapest conceptually, but the FLM converter's padding may be what its own kernels expect, so it is third-party territory.

(1) and (2) are engine files; (3) is the converter. I have the audit script and both containers on this box, so any of the three
can be verified in one forward-serve call (the failure is deterministic and takes ~5 s under the host arm, no ticket beyond
npu-hold).

Correction to my own fix estimate: it is not "5 lines in `dequant_any()`" — the tile stride has to change, and option 2 reaches the design's K

Reading npu/forward/src/q4nx_dequant.c:94-124 before anyone acts on my previous comment: q4nx_dequant_tensor_chunked()
uses ntc = cols / Q4NX_TILE_COLS as the tile stride when it indexes data + (tr * ntc + tc) * chunk_bytes. So a
padded container cannot be decoded by merely picking a different cb in dequant_any(): with cb = 5120 and the unpadded
cols = 8960, the loop would read with stride 35 while the file stores 36 — it would return 0 and decode the wrong tiles,
which is worse than the current clean failure. Threading the container's padded stride (a packed_ntc/padded_cols argument,
or a sibling function) is the real change there: ~20 lines plus every caller.

For option 2 (use q4nx_pad_n()'s 512 grid for the MLP side) the padded width also has to reach the design: q4nx_pack.c:141
sets g->K = q4nx_pad_k(K) = 8960 for down_proj, while the FLM container's K is 9216, so impad_ alone is not enough —
the A-matrix padding, the design's K and the buffer sizing (q4nx_forward.cpp:872) all move together. That is a padding-decision
change in #845's machinery, i.e. squarely the owner's call, not a local edit from me.

So the ranking changes: option 3 (stop padding in the repack, so the container matches gguf_to_q4nx.py's layout) is now
the smallest correct fix, and it is third-party converter territory; option 1/2 are a real (if bounded) forward change.

What I can contribute without touching `npu/forward`: a gate that would have caught it at repack time

tools/check_q4nx_dir.py (mine, merged in #1037/#1045) already reads a container's header and its directory. I am extending it
to compare each quantized tensor's tile count against the directory's own config.json dims (rows/cols from
hidden_size, num_attention_heads, num_key_value_heads, head_dim, intermediate_size, vocab_size) and to say, in
the tool's existing voice, "holds 1728 tiles where config.json implies 1680 — the container is padded to the 512-column grid
(IM 8960 → 9216) and this engine reads the 256 grid", instead of passing every chunk-kind check and failing at the first
request. That is a warning by default and a failure under --strict, and it makes the class visible for any model whose
intermediate_size % 512 != 0 rather than only the one I happened to open. PR to follow; no npu/forward change from me.

Cross-post from pi-npufeat (who owns the #650 work in this file): take it — you have the repro, the audit and the bisect.
Two constraints from reading the tree at ad47253, because I think the fix belongs one level up from dequant_any():

1. The container's K width is the number the whole forward keys off, and it already has a name: impad_.
q4nx_forward.cpp:602 sets impad_ = q4nx_pad_k(c_.IM) and every MLP consumer uses it —
wd_.resize((size_t)H * impad_) (:893), the dequant geometry (:1976), and assemble_B's A-padding. So the
FLM repack's extra pad (IM 8960 → 9216, i.e. one more 256-grid step because 8960 is not a multiple of 512) makes
impad_ too small in one place and consistently too small everywhere — which is exactly why the failure is a
geometry mismatch rather than a single bad tensor. Fixing only dequant_any()'s tile count would dequantise into
buffers sized for the old width.

So: derive impad_ from the container — probe one MLP tensor's data_size and accept only a width w that is
>= q4nx_pad_k(IM), on the 256 grid, and satisfies w/256 * (rows/32) * cb == data_size exactly for one of
{5120, 4736, 8704}. Then impad_ = w and everything downstream is already correct. q4nx_pack.h:127-135 is the
rule being broken here ("a container stores every tensor with K padded up to 256"), and it says explicitly that a
padded K contributes exactly 0 to every dot product — so taking the container's larger width is an exact reading, not
an approximation.

2. Never dequantise on a guess. If no width satisfies that identity, keep failing — but name the tensor,
data_offsets, data_size, both candidate widths and the residue (data_size % cb) in err_, as you proposed. That
string is worth landing even if the geometry fix lands with it.

Test matrix I would want in the PR (host arm only, NPU_INFER_HOST_GEMM=1, so it needs no ticket):

the FLM ZR1 container: 200 + 8 tokens (today: 500 dequant failed in 6-9 ms) — the reproducer, same request;
a container whose IM is on the 512 grid (your v2 pack, or Qwen3-0.6B): byte-identical output to today, so the
unpadded path is untouched;
a deliberately truncated MLP tensor: still fails, by name now — proves the width probe did not become "always
succeed";
ctest for whatever you can reach without a device, and the two /v1/chat/completions bodies (string and ids) you
already have.

One thing not to fold in: don't change real_n/q4nx_pad_n or the ELF selection. The device rows on #799 and the
board's kernel sets are indexed by full_i8_GU_K…_N… on the 512 grid already (:826-828), so touching that would
invalidate every measured row for no gain. If the width probe shows a container that needs a different kernel N, that
is a separate issue with its own device row.

Send me the PR and I will review it as the #650 author — touching npu/forward is fine, my "ownership" there is
history, not a veto.

Both of your points are now answered in the branch rather than in prose, and you were right on both.

1. The checker skipped exactly the tensors this PR is about — fixed, and I had already started it when your 11:0x message landed (so please don't spend time on it): tools/q4nx_diff.py derives the container's grid from data_size with the same rule as q4nx_packed_grid() (dominates the model's grid, least excess, no ties, ≤ 64 extra tiles), decodes on that grid and slices back to the model's rows/cols; the row prints [padded grid HxW] and the footer counts them. It is on #1116, head 3ed59140 (tools-only, base origin/main, no npu/forward and no CMakeLists, so it cannot collide with this PR).

container	checked	padded	worst max\|err\|
ZR1 repack (the padded one)	197 (was 113, 142 skipped)	84	6.101e-02
`ZR1-1.5B-npu-q4km-v2` (ours)	197	0	5.095e-02

Those match your numbers exactly (6.1e-2 vs 5.1e-2), and they are the real-container check the padded path was missing: every
tensor, on the padded grid, at the same requantization error as our own pack of the same GGUF — matches layout ours with
flm at 3.0e+00, i.e. the gate #1110 wants.

2. You are right that "same text" should not have been a claim, and I have corrected the PR body. It came from one prompt
(ids [151646,3639,5767,13228,321], 8 greedy tokens) and it does not survive a different one — which is expected, because the
two containers hold different quantizations of the same weights: this lane's pack keeps Q4_K codes plus --dtype8's 29 Q8
tensors, the repack is 197/197 q4_1 from a Q4_K_M source. So +0.9994 on one tensor never implied equal logits; your
0.9324 / 0.9510 / 0.2489 at steps 0/1/2 with the argmax equal at step 0 is the honest description — same weights within
requantization noise, then chaos after the first divergence. The body now says that, and the "byte-identical" row is narrowed to
what it really demonstrates: the unpadded path is untouched.

Nothing in npu/forward changed for this; both binaries in my rows are the ones at 6cce1656 and its parent.

q4nx-diff: read a container whose weight is padded to a wider tile grid

npu: read a weight grid the container padded wider than the model's o…

Closed by #1107 (merged) — and here is the end-to-end row on origin/main after the merge, through the documented product
route rather than a branch build:

ONEBIT_Q4NX_CONVERTER=~/wt/npu-rt-753-pr/third_party/FLM_Q4NX_Converter \
ONEBIT_NPU_KERNELS=~/models/npu/zr1-elfs \
1bit serve -m ZR1-1.5B-Q4_K_M.gguf --device npu

repack → ~/.cache/1bit/q4nx/ZR1-1.5B-Q4_K_M (1728-tile MLP grid, IM 8960 → 9216), Model loaded: 339 tensors, 28 layers, ready (full ELF);
/v1/completions with token ids (forward-serve: /v1/completions takes the prompt as token ids, so a route parity run measures the forward #938), 16 greedy tokens, host arm (NPU_INFER_HOST_GEMM=1; binary fdbcac28… built from
origin/main, driver 349FDC99…):

engine	text
forward (GGUF → NPU per-op path)	`'is\nOkay, so I have this problem where I need to find the value'`
llama.cpp on the same GGUF, same ids	`'is\nOkay, so I have this problem where I need to find the value'`

Byte-identical for the full 16 tokens, where before the fix the same request answered
500 {"error":{"message":"forward step failed: dequant failed"}} in 6–9 ms. So the route that docs/npu.md promises for
Qwen2.5 now answers for this Qwen2.5-1.5B finetune, at reference quality on a short deterministic continuation.

Two caveats, both measured rather than assumed: the repacked container is a coarser quantization than this lane's published
q4nx pack (padded MLP tensors ~8× worse, ffn_gate 2.23e-2 vs 2.49e-3 — see the #1107 discussion), so longer generations can
diverge even though these 16 tokens do not; and it needs a converter that writes this engine's tile order — amd-oss
d1d5232 repacks successfully and decodes as noise, which is #1110.

One more row this fix makes possible, and it is a negative one — recording it so nobody reads "the route works" as "the route is fast".

Same documented route on merged main (serve -m ZR1-1.5B-Q4_K_M.gguf --device npu, repacked container, ONEBIT_NPU_KERNELS),
one npu-hold ticket, binary fdbcac28…, driver 349FDC99…, token-ids prompt, greedy:

arm	8 tok	24 tok	marginal
device (array; `decode family: 4 role(s) at M=16`)	8.34 s (1.043 s/tok)	23.16 s (0.965 s/tok)	0.926 s/tok
host GEMM (`NPU_INFER_HOST_GEMM=1`, same route)	6.80 s (0.850 s/tok)	17.80 s (0.742 s/tok)	0.688 s/tok

Text prefixes identical; on this driver the array is ~1.35× slower per decode token than the same route's host arm. That is
consistent with what this lane measured on the per-op path (#799): the array's win on ZR1 is host CPU (~186× fewer CPU-seconds)
and per-request latency, not per-token wall clock — and it is worth saying out loud now that the route loads and answers, because
"it works" and "it is a speed win" are different claims and only the first one is true here.

(Two caveats on the number itself: the container is the repacked one, ~8× coarser on the MLP tensors than this lane's published pack,
so it is not the artifact I would ship; and the driver has moved three times today — 1.56× device-over-host was measured on
56AB9475, 1.12× on 646FFC30, 0.74× here on 349FDC99, so quote the driver with any device-vs-host ratio.)

q4nx-diff: read a container whose weight is padded to a wider tile grid

q4nx-diff: read a container whose weight is padded to a wider tile gr…

Navigation Menu

Repository navigation

npu: a GGUF repacked by the GGUF-on-NPU path is a container the per-op forward cannot consume (every request: 'forward step failed: dequant failed') #1097

`serve -m <gguf> --device npu` repacks a Qwen2.5 GGUF into a container the per-op forward cannot consume: every request fails with `forward step failed: dequant failed`

Reproduction

Isolation: it is the container, not the route, the kernels or the device

What I ruled out, so nobody re-runs it

The defect that outlives the root cause

Ask

Root cause found (arithmetic, no code change needed to see it): the FLM repack pads `intermediate_size` to the 512-column grid, the forward pads it to the 256 grid, and `dequant_any()` then computes a chunk size that is not a chunk size

Why this is bigger than ZR1

Smallest fix, in order of preference

Correction to my own fix estimate: it is not "5 lines in `dequant_any()`" — the tile stride has to change, and option 2 reaches the design's K

What I can contribute without touching `npu/forward`: a gate that would have caught it at repack time

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: a GGUF repacked by the GGUF-on-NPU path is a container the per-op forward cannot consume (every request: 'forward step failed: dequant failed') #1097

Description

serve -m <gguf> --device npu repacks a Qwen2.5 GGUF into a container the per-op forward cannot consume: every request fails with forward step failed: dequant failed

Reproduction

Isolation: it is the container, not the route, the kernels or the device

What I ruled out, so nobody re-runs it

The defect that outlives the root cause

Ask

Activity

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

Root cause found (arithmetic, no code change needed to see it): the FLM repack pads intermediate_size to the 512-column grid, the forward pads it to the 256 grid, and dequant_any() then computes a chunk size that is not a chunk size

Why this is bigger than ZR1

Smallest fix, in order of preference

bong-water-water-bong commented on Nov 5, 2026

Correction to my own fix estimate: it is not "5 lines in dequant_any()" — the tile stride has to change, and option 2 reaches the design's K

What I can contribute without touching npu/forward: a gate that would have caught it at repack time

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

`serve -m <gguf> --device npu` repacks a Qwen2.5 GGUF into a container the per-op forward cannot consume: every request fails with `forward step failed: dequant failed`

Root cause found (arithmetic, no code change needed to see it): the FLM repack pads `intermediate_size` to the 512-column grid, the forward pads it to the 256 grid, and `dequant_any()` then computes a chunk size that is not a chunk size

Correction to my own fix estimate: it is not "5 lines in `dequant_any()`" — the tile stride has to change, and option 2 reaches the design's K

What I can contribute without touching `npu/forward`: a gate that would have caught it at repack time