[bong-water-water-bong]

Adds tools/gguf_to_q4nx.py. It repacks a qwen3 GGUF into a Q4NX model directory (model.q4nx, config.json, and a tokenizer copied from --tokenizer), so a GGUF can run on the NPU.

• Every projection and lm_head is written as q4_1 tiles, the kind the lane and dx read.
• Q4_0, Q4_1 and Q4_K map code for code. They are exact apart from rounding scale and zero to bf16; a Q4_K super-block is one tile row's 256 columns.
• Q5_K, Q6_K, Q8_0 and float tensors are dequantized and requantized to q4_1, which is lossy.
• verify compares every tensor with the GGUF, dequantized. The exact types must stay within the bf16 rounding bound; for the requantized ones it reports the error.
• tests/gguf_to_q4nx_test.py (runs in CI) checks the tile layout in both directions against the committed golden q4_1 tile, which has a verified SHA-256. It also packs and verifies a synthetic GGUF with Q4_0, Q4_1, Q4_K, Q8_0 and F32 tensors.

Measured on strixhalo with Qwen3-0.6B-Q4_K_M.gguf:

• Repack: all 311 tensors pass verify. The Q4_K tensors have 0 weights over the bound. The Q6_K tensors requantize to cosine 0.9969.
• dx route: dx_decode_test --ref against dx_model.py ref on the repacked directory gives 24/24 tokens, logits cosine at least 0.99995, 21.0 ms/step, and 0 TDRs.
• 1bit serve --device npu: loads in 1.1 s and answers "The capital of France is Paris." at 59.6 tok/s.

Limits: qwen3 only. The GGUF vocabulary is not converted into tokenizer.json.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit aa04259

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit aa04259)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Precision Loss in Q4_K Handling The handling of Q4_K tensors in split_q4_k function may introduce precision loss due to the conversion of scale and zero points to bf16. While the documentation states that the conversion is exact apart from rounding to bf16, the specific rounding behavior (nearest even) and the potential for accumulation of rounding errors across multiple tiles could lead to subtle differences in the final model output. This is particularly concerning for the dx route on NPU where precision is critical. def split_q4_k(raw, N, K): b = raw.reshape(N, K // 256, 144) d = f16(b[..., 0:2])[..., 0][..., None] dmin = f16(b[..., 2:4])[..., 0][..., None] s = b[..., 4:16].astype(np.uint16) sc = np.empty(s.shape[:-1] + (8,), np.uint16) mn = np.empty_like(sc) sc[..., 0:4] = s[..., 0:4] & 63 mn[..., 0:4] = s[..., 4:8] & 63 sc[..., 4:8] = (s[..., 8:12] & 0xF) | ((s[..., 0:4] >> 6) << 4) mn[..., 4:8] = (s[..., 8:12] >> 4) | ((s[..., 4:8] >> 6) << 4) qs = b[..., 16:144].reshape(N, K // 256, 4, 32) # 4 chunks of 64 values codes = np.stack([qs & 0xF, qs >> 4], axis=3).reshape(N, K) # low nibbles, then high scale = (d * sc.astype(np.float32)).reshape(N, K // 32) # exact in f32 (11 x 6 bits) zero = (-dmin * mn.astype(np.float32)).reshape(N, K // 32) return codes, scale, zero Incorrect Tensor Shape Handling In the to_q4_1 function, the tensor shape is interpreted as K, N (where K is columns and N is rows) but the code assumes N is the number of rows and K is the number of columns. This could lead to incorrect tensor processing if the tensor dimensions are not as expected, especially when dealing with different tensor layouts in GGUF files. K, N = int(t.shape[0]), int(t.shape[1]) # ggml order: ne0 = columns if N % ROWS or K % COLS: raise SystemExit(f"{t.name}: {N} x {K} is not a whole number of 32 x 256 tiles") split = EXACT.get(t.tensor_type) if split: return (*split(np.asarray(t.data).reshape(-1), N, K), True) return (*quantize_q4_1(dequantize(t.data, t.tensor_type).reshape(N, K).astype(np.float32)), False) Memory Mapping Usage in Verify The verify function uses np.memmap to read the model file, which is efficient for large files. However, the code does not explicitly handle potential memory mapping errors or ensure that the file is not modified during the verification process. This could lead to inconsistent or incorrect verification results if the file is accessed concurrently. def verify(args): """Every Q4NX tensor against the GGUF, dequantized.""" r = gguf.GGUFReader(args.gguf) with open(os.path.join(args.out, "model.q4nx"), "rb") as f: n = struct.unpack("<Q", f.read(8))[0] hdr = json.loads(f.read(n)) base = 8 + n mm = np.memmap(os.path.join(args.out, "model.q4nx"), dtype=np.uint8, mode="r")

[github-actions]

Persistent review updated to latest commit aa04259