[bong-water-water-bong]

Root cause

tools/check_q4nx_dir.py states its own limit: two 8704-byte Q8 chunks share a shape, "the
container does not record which one it holds - so it reports the Q8 tensors and says so, rather
than guessing". That gap is silent by construction: a container whose codes are in the other
order decodes to plausible floats, so it serves noise with no error anywhere. --dtype8 (#960)
made Q8 chunks a container feature, and the only way anyone could confirm the order was a
hand-written script that is not in the tree.

The check cannot come from the packer's round trip: encoder and decoder can agree with each other
and both disagree with the weights. It has to come from the model.

Fix

check_q4nx_dir.py --against <source.gguf>: for each Q8 tensor (first N, default 3) read its first
tile, decode it in both code orders, and correlate each against the same tensor in the GGUF the
container was packed from — matched through the packer's own hf_name() mapping, so there is one
name mapping in the tree, not two. Pass = the order npu/q4nx.cpp decodes wins at corr ≥ 0.99 and
rel-rms ≤ 0.10; otherwise exit 1 with the tensor's name and both scores. The default path is
unchanged: metadata only, no numpy, no gguf, nothing dequantized. numpy/gguf are imported inside
the new mode only.

Measured — ZR1-1.5B, the shipped 1bit-MONSTER/ZR1-1.5B-GGUF container (29 Q8 tensors)

Source: ZR1-1.5B-Q4_K_M.gguf (9457f6ba…), tile (0,0) of each tensor. Layout A is
column*32 + row (what npu/q4nx.cpp decodes), B is the Qwen3.6-35B order.

tensor	in the GGUF	A corr / rel-rms	B corr
lm_head.weight	output.weight	+1.0000 / 0.0053	+0.0324
model.layers.0.self_attn.v_proj.weight	blk.0.attn_v.weight (Q6_K)	+1.0000 / 0.0054	+0.0009
model.layers.0.mlp.down_proj.weight	blk.0.ffn_down.weight	+1.0000 / 0.0055	+0.0250

The 0.005 is Q6_K → Q8 requantization, not a layout error. Before/after on the same container with
one tile's codes moved to B (scales untouched, i.e. the silent failure):

before: chunk layout verified ...                              rc 0
after : FAIL model.layers.0.self_attn.v_proj.weight: not the layout npu/q4nx.cpp decodes
        (A corr -0.0044 rel-rms 1.4519, B corr +1.0000)        rc 1

Tests

Three rows in tests/gguf_to_q4nx_test.py, against the synthetic GGUF that test already builds
(so this runs in the existing CI step, no new dependency, no model on disk):

• --against accepts the container the packer just wrote — rc 0;
• --against refuses the same container with one Q8 tile permuted into the other order — rc 1;
• the refusal names the tensor.

Plus a --self-test row that builds a tile in one order and asserts the two decoders disagree about
it, so the verdict cannot come from a decoder agreeing with itself. --self-test and
tests/gguf_to_q4nx_test.py both pass; tools/copyright.py --check clean.

Not in this PR: making --self-test run in CI (it needs a workflow-scoped token to edit
.github/workflows/ci.yml); the rows above already run through tests/gguf_to_q4nx_test.py.

Related: #960 (adds the --dtype8 chunks this verifies), #799 (plan/kernel-layer Q8 notes).
No issue is closed by this — it is the missing half of #960's own verification story.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 2fad106)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 960 - Partially compliant Compliant requirements: The PR introduces the --dtype8 flag for gguf_to_q4nx.py to store tensors as Q8 chunks The verify process now checks Q8 tensors against source GGUF for accuracy q4nx_to_gguf.py writes back Q8 tensors as GGML Q8_0 Documentation is updated with new measurements and type table rows Non-compliant requirements: The PR does not include the dx route changes to serve --dtype8 containers (mentioned in the ticket as a separate change) The PR does not include the device decode changes for Q8 chunks (mentioned in the ticket as a separate change) Requires further human verification: Verification that the dx route can actually serve --dtype8 containers (mentioned as a separate change) Verification that the device decode of Q8 chunks works correctly (mentioned as a separate change)

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Q8 Layout Verification Logic The new --against functionality in check_q4nx_dir.py attempts to verify Q8 tile layouts by comparing against source GGUF tensors. However, the correlation threshold (corr >= 0.99) and rel-rms threshold (rel-rms <= 0.10) may be too strict for some valid cases, potentially causing false positives. The check should be validated with actual model data to ensure it doesn't reject valid containers. if not (a[0] >= 0.99 and a[1] <= 0.10 and a[0] > b[0]): errors.append(f"{name}: not the layout npu/q4nx.cpp decodes " f"(A corr {a[0]:+.4f} rel-rms {a[1]:.4f}, B corr {b[0]:+.4f})") Test Coverage for Q8 Layout The test suite now includes checks for Q8 tile layout verification, but the test for permuted Q8 tiles may not cover all edge cases. Specifically, the permutation logic in permute_q8_tile function needs to be verified to ensure it correctly creates a tile with the alternative code order, and that the verification correctly identifies when a container uses the wrong layout. def permute_q8_tile(path, tensor): """Rewrite one Q8 tensor's first tile in the OTHER 8704-B code order (Qwen3.6-35B's). The scales stay where they are, so the tile still decodes to numbers -- which is the failure this test is about: a container in the wrong order serves noise, not an error. """ import json import struct with open(path, "r+b") as f: n = struct.unpack("<Q", f.read(8))[0] hdr = json.loads(f.read(n)) base = 8 + n off = hdr[tensor]["data_offsets"][0] f.seek(base + off) tile = bytearray(f.read(G.Q8)) codes = bytes(tile[512:]) moved = bytearray(8192) for r in range(32): for k in range(256): moved[(r // 16) * 4096 + (k // 32) * 512 + (k % 32) * 16 + (r % 16)] = codes[k * 32 + r] tile[512:] = moved f.seek(base + off) f.write(tile)

[bong-water-water-bong]

VERDICT: READY at b1d31692

The claim is that the tool now decides a Q8 tile's code order against the model rather than assuming it, and that is exactly what it does. I ran it on a real --dtype8 container against its real source GGUF rather than only on the test's fixtures:

$ python3 tools/check_q4nx_dir.py ~/models/npu/ZR1-1.5B-npu-q4km-q8 \
      --against ~/models/zyphra-zr1/ZR1-1.5B-bf16.gguf
  lm_head.weight                 vs output.weight      layout A: corr +0.9998 rel-rms 0.0187 | B: corr +0.0321 rel-rms 1.4363
  blk.0.self_attn.v_proj.weight  vs blk.0.attn_v.weight layout A: corr +0.9998 rel-rms 0.0188 | B: corr +0.0011 rel-rms 1.4447
  blk.0.mlp.down_proj.weight     vs blk.0.ffn_down.weight layout A: corr +0.9998 rel-rms 0.0193 | B: corr +0.0247 rel-rms 1.4302
  -> chunk layout verified, and the Q8 tiles hold the code order npu/q4nx.cpp decodes   (exit 0)

29 Q8 tensors in that container, so this is the case the PR is about, and the separation is not marginal: the correct layout correlates at 0.9998 while the other lands at 0.001-0.03 — i.e. the "serves noise with no error anywhere" failure is now a two-number comparison. The acceptance gate is stated in the code and is the right shape (a[0] >= 0.99 and a[1] <= 0.10 and a[0] > b[0], so a container that is right must also be better than the alternative, which a both-wrong decode cannot fake).

Also checked:

• python3 tests/gguf_to_q4nx_test.py → PASS, rc=0, including --against refuses the other code order (rc 1) and the refusal names the tensor — so the negative is in the suite, not only in the prose;
• a container with no Q8 tensors is handled honestly rather than vacuously: ~/models/npu/Qwen3-0.6B reports no Q8 tensors, nothing to check against the source weights and still exits 0 with its chunk table verified (197 q4_1 tensors), which is the right behaviour for the many packs that have none;
• tools/copyright.py --check → rc=0;
• the write-back case in the test (the Q8 container writes back as 2 Q8_0 tensors (got 2)) covers the other direction, which is the one the packer could get wrong while the decoder stays right.

One nit: the thresholds are in the code but not in the help text, and a user who sees corr +0.90 needs to know what the tool will call it. One line in --help (passes when correlation >= 0.99, rel-rms <= 0.10, and the declared order beats the other) makes the printed table self-explanatory, and it is the number a future packer change should be judged against.

Not re-run: nothing device-side — this tool is pure host code over the container and the GGUF, which is why the check above is the whole claim.

[github-actions]

Persistent review updated to latest commit d468141

[github-actions]

Persistent review updated to latest commit 2fad106

[bong-water-water-bong]

VERDICT: READY at d4681414

This push closes the hole I flagged and it does so on the right case. Every mode, run just now:

Q8 container + the WRONG gguf  -> rc=1, "chunk layout NOT verified"
    FAIL ... v_proj.weight: not the layout npu/q4nx.cpp decodes (A corr -0.0140 rel-rms 1.6072, B corr -0.0196)
    FAIL ... down_proj.weight: not the layout npu/q4nx.cpp decodes (A corr +0.0198 rel-rms 1.7021, B corr -0.0186)
Q8 container + its own gguf    -> rc=0, "chunk layout verified, and the Q8 tiles hold the code order npu/q4nx.cpp decodes"
    (A corr +0.9998 rel-rms 0.0193 for down_proj, B +0.0247 rel-rms 1.4302)
container with no Q8 tensors   -> rc=0, "no Q8 tensors, nothing to check against the source weights"

The middle row is the one that matters: a wrong GGUF now lands at rc=1 naming the tensor, where before it would have compared nothing and printed success. rows non-empty is the right guard, and except (SystemExit, KeyError, ValueError): continue for a GGUF entry with no Q4NX name is correct — that entry is not ours to check, so skipping it cannot turn into a false negative, whereas a blanket "skip anything that fails to map" could.

Two small things:

1. chunk layout verified, and the Q8 tiles hold the code order npu/q4nx.cpp decodes is printed in the no-Q8 case too, where no order was established. It is true (the chunk layout is verified, and there are no tiles to be wrong), but the sentence reads as though a code order was checked. Something like chunk layout verified; no Q8 tensors to check there would be exact.
2. The acceptance thresholds are still not in --help (corr >= 0.99, rel-rms <= 0.10, and the declared order must beat the other). A user reading A corr -0.0140 in the FAIL line above knows it failed, not what it had to reach.

The device-free claim is unchanged and still the whole story here — this tool never touches the GPU.