Skip to content

npu: check_q4nx_dir calls every gated (attn_output_gate) pack a padded grid — the real d3 pack and all seven 27B qwen3_5 dirs fail --strict #1285

Closed

Description

@bong-water-water-bong

What

tools/check_q4nx_dir.py derives every weight's expected tile grid from config.json and warns when a
container's tile count disagrees. For self_attn.q_proj.weight it expects num_attention_heads × head_dim
rows — but with attn_output_gate the stored q_proj emits q and the output gate, so it holds
2 × that many rows. Every gated pack on this box is therefore reported as a converter-padded grid:

WARN model.layers.3.self_attn.q_proj.weight: holds 512 tiles, the model's dimensions imply 256 -
columns padded 1024 -> 2048 (the 512-column grid); this engine derives the chunk size from the unpadded
grid, so the tensor fails to decode (engine#1097)

and --strict exits 1, i.e. the checker tells an operator that a correct container is broken — the exact
inverse of what it exists for. Found while closing out #796 (the real vllm-sr/d3-lite pack).

Measured (strixhalo, header reads only, no device)

tools/check_q4nx_dir.py <dir> --strict over all 18 Q4NX model directories on this box, before → after:

container attn_output_gate WARN before WARN after --strict before after
d3-lite-conv (the real d3 pack) true 6 (the 6 full-attention layers) 0 1 0
bonsai2-27b-NPU, -NPU-b8, -pb16i, -pb16i-unrot, -t2s2, -t2s2-serve 1 16 each 0 1 0
bonsai2-t2s2-perrole876 1 16 0 1 0
the other 10 dirs (Qwen3-0.6B ×3, ZR1-1.5B ×4, d3-lite-bf16, d3-synth-0.6b, qwen3-coder-30b-a3b-moe) absent 0 0 0 0

So the false positive covers 8 of 18 containers here, including the production 27B serve directory, and
nothing that was clean before changes.

The container is right, and the checker is wrong (two independent checks)

  • The tensor agrees with 2*QD. d3-lite-conv's model.layers.3.self_attn.q_proj.weight is
    I8 [512, 5120], i.e. 512 tiles of 32×256 = 4096 rows (4096/32 × 1024/256 = 512), against
    QD = 8 × 256 = 2048. The container drops 2 × 2048 = 4096 rows out of q4nx_derive_designs' own
    number: its plan.json layer 3 fused QKV is K=1024 N=5120, and N - 2*KVD = 5120 - 2*512 = 4096 is
    exactly the q|gate pair.
  • It decodes and answers. The same container's route returns 10/10 decisions on the array and 10/10
    under NPU_INFER_HOST_GEMM=1 (npu: a real d3 pack (vllm-sr/d3-lite, 0.8B-GDN) answering through /v1/systemone — converter transforms, name map, and the 7-design ELF set #796's rungs 1 and 2), so those six tensors cannot be "failing to decode".

Fix

dims_from_config carries QG = 2*QD when config.json sets attn_output_gate (JSON true or 1 —
the forward reads it with q4nx_config_int), and expected_rc uses QG for q_proj. Two files, 26+/4−:
the tool, a self-test case, and one sentence in docs/npu.md where the checker is described.

Verification

  • tools/check_q4nx_dir.py --self-test (ctest q4nx_dir_check) passes, including the new case: a d3
    q_proj of [512, 5120] tiles is quiet under gated dims and still warns under ungated dims, so the case
    can only pass when the gate is what suppresses the warning.
  • Tested the test: with qg = qd restored, the self-test fails on
    attn_output_gate did not double q_proj's rows: {'QD': 2048, 'QG': 2048, …}; restored, it passes.
  • The sweep above (18 dirs, old vs new script side by side) and --strict rc on the two gated families.
  • A freshly converted container (safetensors_to_q4nx.py pack --arch qwen3_5 on the 1.7 GB pack):
    weight bytes byte-identical to the shipped container (sha256 313ac4f9…), verify 320/320 ok, old
    checker 6 WARN / rc 1 → new checker 0 WARN / rc 0.

PR to follow.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions