Repository navigation
Description
What
tools/check_q4nx_dir.py derives every weight's expected tile grid from config.json and warns when a
container's tile count disagrees. For self_attn.q_proj.weight it expects num_attention_heads × head_dim
rows — but with attn_output_gate the stored q_proj emits q and the output gate, so it holds
2 × that many rows. Every gated pack on this box is therefore reported as a converter-padded grid:
WARN model.layers.3.self_attn.q_proj.weight: holds 512 tiles, the model's dimensions imply 256 -
columns padded 1024 -> 2048 (the 512-column grid); this engine derives the chunk size from the unpadded
grid, so the tensor fails to decode (engine#1097)
and --strict exits 1, i.e. the checker tells an operator that a correct container is broken — the exact
inverse of what it exists for. Found while closing out #796 (the real vllm-sr/d3-lite pack).
Measured (strixhalo, header reads only, no device)
tools/check_q4nx_dir.py <dir> --strict over all 18 Q4NX model directories on this box, before → after:
| container | attn_output_gate |
WARN before | WARN after | --strict before |
after |
|---|---|---|---|---|---|
d3-lite-conv (the real d3 pack) |
true | 6 (the 6 full-attention layers) | 0 | 1 | 0 |
bonsai2-27b-NPU, -NPU-b8, -pb16i, -pb16i-unrot, -t2s2, -t2s2-serve |
1 | 16 each | 0 | 1 | 0 |
bonsai2-t2s2-perrole876 |
1 | 16 | 0 | 1 | 0 |
| the other 10 dirs (Qwen3-0.6B ×3, ZR1-1.5B ×4, d3-lite-bf16, d3-synth-0.6b, qwen3-coder-30b-a3b-moe) | absent | 0 | 0 | 0 | 0 |
So the false positive covers 8 of 18 containers here, including the production 27B serve directory, and
nothing that was clean before changes.
The container is right, and the checker is wrong (two independent checks)
- The tensor agrees with
2*QD.d3-lite-conv'smodel.layers.3.self_attn.q_proj.weightis
I8 [512, 5120], i.e. 512 tiles of 32×256 = 4096 rows (4096/32 × 1024/256 = 512), against
QD = 8 × 256 = 2048. The container drops2 × 2048 = 4096rows out ofq4nx_derive_designs' own
number: itsplan.jsonlayer 3 fused QKV isK=1024 N=5120, andN - 2*KVD = 5120 - 2*512 = 4096is
exactly the q|gate pair. - It decodes and answers. The same container's route returns 10/10 decisions on the array and 10/10
underNPU_INFER_HOST_GEMM=1(npu: a real d3 pack (vllm-sr/d3-lite, 0.8B-GDN) answering through /v1/systemone — converter transforms, name map, and the 7-design ELF set #796's rungs 1 and 2), so those six tensors cannot be "failing to decode".
Fix
dims_from_config carries QG = 2*QD when config.json sets attn_output_gate (JSON true or 1 —
the forward reads it with q4nx_config_int), and expected_rc uses QG for q_proj. Two files, 26+/4−:
the tool, a self-test case, and one sentence in docs/npu.md where the checker is described.
Verification
tools/check_q4nx_dir.py --self-test(ctestq4nx_dir_check) passes, including the new case: a d3
q_projof[512, 5120]tiles is quiet under gated dims and still warns under ungated dims, so the case
can only pass when the gate is what suppresses the warning.- Tested the test: with
qg = qdrestored, the self-test fails on
attn_output_gate did not double q_proj's rows: {'QD': 2048, 'QG': 2048, …}; restored, it passes. - The sweep above (18 dirs, old vs new script side by side) and
--strictrc on the two gated families. - A freshly converted container (
safetensors_to_q4nx.py pack --arch qwen3_5on the 1.7 GB pack):
weight bytes byte-identical to the shipped container (sha256 313ac4f9…),verify320/320 ok, old
checker 6 WARN / rc 1 → new checker 0 WARN / rc 0.
PR to follow.
Activity