[bong-water-water-bong]

P2 follow-up to #809 (t2 chunk kind). Two questions were asked on #801 by the consumers, and both
change what a reader has to do, so the answers belong in the spec and not only in a comment thread.

1. A tensor that is not whole tiles is zero-padded (pinned)

K % 256 == 0 and N % 32 == 0 were stated as geometry and nothing said what to do when a tensor does
not have it. The pinned convention: pad with code 0 and scale 0.0, i.e. a weight of exactly 0 —
so a padded K column contributes exactly 0 to the reduction whatever the activation is, and a padded N
row is never read. The chunk has no field for the original shape, so the padded shape is the tensor's
shape.

• tools/q4nx_t2.py: new pad_to_tile(); pack(..., pad=False) still refuses a non-aligned tensor by
  default; from_ptq1_0 / from_tq1_0 / from_tq2_0 pass pad through. (from_pb16i is untouched —
  it is tile-native.)
• All 431 ternary Bonsai-2 images are already whole tiles (5120 / 6144 / 17408-column rows, measured), so
  this exists for GGUF tensors and for a projection whose width is not a multiple of 32.
• tests/q4nx_t2_test.py asserts the convention, not the size: the real block bit-exact after padding,
  the padding decoding as exactly 0, a padded column adding nothing to a dot product, and the padded
  scale slots themselves zero (a padder that leaves them at 1.0 is caught — that check was verified by
  breaking pad_to_tile in place).

2. The stored weight is the folded (post-rotation) one; t2 carries no rotation

docs/npu.md now states it with the code that proves it: a folded container's W' already has the block
Hadamard and the per-width signs baked in, and the consumer rotates the activation into that basis —
A' = H (A . s), signs first, then the normalized Sylvester-Walsh butterfly, then 1/sqrt(block)
(npu/forward/src/q4nx_forward.cpp::hadamard_rotate) — never the weight. So the chunk needs no
pre/post-rotation flag and no sign vector: the fold stays declared where it belongs, in the container
(q4nx_hadamard, q4nx_hadamard_block, the HAD1 sign table). Reading a t2 tensor in the wrong basis is
a consumer bug, not a format one. The inverse direction (the tied embedding, inverse_weight_names) is
S (H y), which round-trips.

Only tools/q4nx_t2.py, tests/q4nx_t2_test.py and docs/npu.md change; no decoder, no kernel, no
container.

Measured gates

python3 tests/q4nx_t2_test.py --no-model      PASS q4nx_t2   (new padding assertions included)
python3 tests/q4nx_t2_test.py                 <full 431-image leg result appended in a comment>
python3 tools/copyright.py --check            rc 0

The real-model leg is the #809 gate unchanged (431 images, whole tensors, bit-exact against the pb16i
dequantisation); this PR does not touch that path, and it was re-run to confirm the modified converter
still produces it.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 41edf86)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 809 - Partially compliant Compliant requirements: Add support for the t2 chunk kind (2-bit ternary + fp16 scale per 128 K, 2.125 b/w) Implement CPU dequantization for t2 chunks Add converters from pb16i, PTQ1_0, TQ1_0, and TQ2_0 formats to t2 Provide bit-exact oracle tests comparing t2 dequantization against pb16i floats Update documentation to describe the t2 chunk kind Non-compliant requirements: No kernel expansion for t2 -> int4 on the core (P3 and P5 ownership) No Q4nxNpuForward wiring for t2 containers No measured tok/s for t2 Requires further human verification: Verification that the on-core expansion for t2 is handled in P3 (XDNA) and P5 (Hexagon) Verification that the MTP head's pb16i-exact set is not t2-convertible

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Padding Convention Implementation The implementation of the padding convention for non-whole tile tensors in tools/q4nx_t2.py correctly pads with code 0 and scale 0.0, ensuring that padded K columns contribute exactly 0 to the reduction. However, the padding logic in pad_to_tile function may not be optimal for performance in cases where large tensors are being padded, as it creates new arrays and copies data, which could be inefficient in a high-frequency inference context. def pad_to_tile(codes, scales): """Zero-pad a tensor to t2 geometry (N -> % 32, K -> % 256). Returns (codes, scales). This is the pinned convention for a tensor whose shape is not whole tiles (docs/npu.md): the padding is code 0 with scale 0.0, so a padded K column is a weight of exactly 0 and contributes 0 to the reduction for any activation, and a padded N row is never read. The caller keeps the padded shape -- the chunk has no field to record the original one. """ codes = np.asarray(codes, dtype=np.int8) scales = np.asarray(scales, dtype=np.float16) N, K = codes.shape if scales.shape != (N, K // GROUP): raise ValueError(f"scales must be [N, K//128] = {(N, K // GROUP)}, got {scales.shape}") Np, Kp = -(-N // ROWS) * ROWS, -(-K // COLS) * COLS if (Np, Kp) == (N, K): return codes, scales c = np.zeros((Np, Kp), np.int8) c[:N, :K] = codes s = np.zeros((Np, Kp // GROUP), np.float16) s[:N, :K // GROUP] = scales return c, s Padding Test Coverage The test suite in tests/q4nx_t2_test.py includes tests for padding behavior, but it only tests a single case (384x30 tensor) and does not cover all possible edge cases for padding, such as tensors that are much larger or smaller than the tile size. This could lead to missed issues in padding logic for edge cases. # 1b. a tensor that is not whole tiles: refused by default, zero-padded with pad=True. The # convention is what a consumer relies on, so assert the arithmetic and not just the size: # the real block is bit-exact, the padding is exactly 0 (code 0 x scale 0), so a padded K # column contributes nothing to a dot product whatever the activation is. Kn, Nn = 384, 30 # neither % 256 nor % 32 (128-wide scale groups, so 384) c_n = rng.integers(-1, 2, size=(Nn, Kn)).astype(np.int8) s_n = (rng.random((Nn, Kn // 128)) * 0.01).astype(np.float16) try: T.pack(c_n, s_n) check(False, "pack() must refuse a tensor that is not whole tiles") except ValueError: pass raw_n = T.pack(c_n, s_n, pad=True) check(len(raw_n) == T.nbytes(512, 32), "pad=True pads 30x300 to the next whole-tile geometry (32x512)") got_n = T.unpack(raw_n, 512, 32) check(np.array_equal(got_n[:Nn, :Kn], c_n.astype(np.float32) * np.repeat(s_n.astype(np.float32), 128, axis=1)), "pad=True leaves the real block bit-exact") check(not got_n[Nn:, :].any() and not got_n[:, Kn:].any(), "the padding decodes as exactly 0 (code 0, scale 0)") codes_p, scales_p = T.pad_to_tile(c_n, s_n) check(not scales_p[Nn:, :].any() and not scales_p[:, Kn // 128:].any(), "the padding carries a zero scale too (0.0, as the format doc says)") act = rng.standard_normal(512).astype(np.float32) check(float(np.abs(got_n @ act - got_n[:, :Kn] @ act[:Kn]).max()) == 0.0, "a padded K column contributes exactly 0 to the reduction")

[bong-water-water-bong]

VERDICT: READY at 3294efd (tprs-ready applied)

Both halves of the PR are a convention, so I checked that the convention is enforced rather than merely written down.

1. Zero padding is enforced, and the check bites. python3 tests/q4nx_t2_test.py --no-model → PASS q4nx_t2.
Then I mutated the padder in place (np.zeros → np.ones for the padded scale slots) and re-ran:

FAIL the padding carries a zero scale too (0.0, as the format doc says)
FAIL q4nx_t2 (1 checks)

so the assertion is load-bearing, not decoration. The other new checks state the arithmetic and not just the size: the
real block stays bit-exact through pack(..., pad=True)/unpack, the padding decodes to exactly 0, and a padded K
column changes a dot product by exactly 0.0. pack() still refuses a non-aligned tensor without pad.

2. The geometry claim is true of the real images: all 431/431 bonsai2-pb16i-ternary images are whole tiles
(size % 4608 == 0 on every file), which is what makes this a GGUF/odd-width feature rather than a change to the
shipped path — and from_pb16i is not touched by the diff.

3. The fold claim matches the code, line by line. hadamard_rotate (npu/forward/src/q4nx_forward.cpp:126)
multiplies the activation by the signs first, then runs the Sylvester-Walsh butterfly per block, then scales by
1/sqrt(block) — exactly the A' = H (A . s) the new doc text states — and it is called on activations only
(:2598, :2823). The container side it names exists: q4nx_hadamard / q4nx_hadamard_block (:414-415) and the
HAD1 sign table (:2491-2501).

Two nits, neither blocking:

• the new test's comment says "pad=True pads 30x300 to the next whole-tile geometry (32x512)" but the fixture is
  Kn, Nn = 384, 30, so it is 30x384 → 32x512. The assertion is right; the comment's 300 is stale.
• inverse_weight_names is named in docs/npu.md as living "in the pack", and that identifier appears nowhere in
  the public tree (only in this new doc line) — the packer is private. Either name the private file or drop the
  identifier; a public doc pointing at a name no public reader can find is the same class as citing a lane state file.

Owed and running, not a blocker: the full 431-image leg (q4nx_t2_test.py --images …, ~6 min). It cannot reach the new
padding code (every image is whole tiles) — it is the unchanged from_pb16i/pack path, and pack's only change for
an aligned tensor is a np.asarray(…, float16) coercion — so I am not holding the verdict on it; I will post the line
when it returns.

[bong-water-water-bong]

The 431-image leg returned, green, on the modified converter:

$ python3 tests/q4nx_t2_test.py --images ~/models/npu/bonsai2-pb16i-ternary
431 images, 25.66 G weights: pb16i 14.44 GB -> t2 6.82 GB (2.1250 b/w), 431/431 bit-exact vs the pb16i floats
PASS q4nx_t2        rc 0        (6 min, one core)

So pad=False's pass-through changed nothing on the real images, and the same numbers the t2 chunk kind reports
(14.44 → 6.82 GB, 2.1250 b/w) still hold at this head. That closes the one item the verdict left open; tprs-ready
stands.

[github-actions]

Persistent review updated to latest commit a8b1963

[github-actions]

Persistent review updated to latest commit 2f76f9b

[github-actions]

Persistent review updated to latest commit 41edf86

[bong-water-water-bong]

VERDICT: READY at 41edf86f1 (head after #816 merged; tprs-ready already on)

Fourth head on this PR, content unchanged and re-checked at it:

• own diff is still tools/q4nx_t2.py (+49/−8), tests/q4nx_t2_test.py (+29/−1), docs/npu.md (+18), and all three
  files are byte-identical to the head I reviewed (3294efd);
• python3 tests/q4nx_t2_test.py --no-model → PASS q4nx_t2 at this head. The 431-image leg is not re-run per head
  because the PR's bytes have not changed since the run that produced 431/431 bit-exact, 14.44 GB → 6.82 GB — that
  result stands for the same tree;
• CI on this head: build, CodeQL, all four Analyze jobs and pr_agent success, hw_gate skipped;
  mergeable_state=clean.

For the record, the two nits from my first verdict are still open on the head that will merge, both cosmetic:
the test comment says "30x300" where the fixture is 30x384, and docs/npu.md names inverse_weight_names "in the
pack" — an identifier that exists only in that sentence in the public tree.