npu-pack: layer norms (and the final norm) are located only for hidden 896

Part of #671 (item 4: a second pack family). Running 1bit npu-pack info on the second pack,
amd/Llama-3.2-1B-Instruct_rai_1.8.0_npu_4K, parses cleanly — model_type: llama, hidden 2048,
intermediate 8192, 32 heads / 8 KV heads, head_dim 64, vocab 128256, rope_theta 500000, 16 layers — and
the head MatMulNBits (N=128256, K=2048, bits 4, block 128) is importable. 145 of its 147 tensors are
refused, and 33 of those refusals are for a reason the first pack never showed:

model.layers.0.input_layernorm.weight -> the .fconst offsets are established for hidden 896 only
model.layers.0.post_attention_layernorm.weight -> (same)
model.norm.weight -> the .fconst tail envelope is [eps][pad][448 f32], not an RMSNorm weight

What is known, and why the constants do not carry over:

In the decode .fconst, layer block L starts at L * 8,708,288 for the Qwen2.5-0.5B pack, with the
bf16 RMSNorm epsilon (86 35) at +0, input_layernorm (bf16, 896 values) at +64, and
post_attention_layernorm at +1,116,032. Confirmed 48/48 on that pack, no hybrid pack needed.
Both offsets are hidden-dependent: the 64-byte header is fixed, but post_attention_layernorm sits
after the attention payload, whose size is (number of 34,816-byte tiles) * 34,816, and the tile count
is sum over q,k,v,o of N/64 (Qwen2.5-0.5B: (896+128+128+896)/64 = 32 tiles, which is what the pack
shows). For Llama-3.2-1B that is (2048+512+512+2048)/64 = 80 tiles, so the offset differs by
80 - 32 tiles, and the block stride differs too.
The per-projection N and K needed to compute those counts are in the pack's own
cache/*_meta.json (op_list[].attrs.N/K for each MladfMatMul, whose const_args are named
<tensor>.preformat), so the numbers are available without the vendor runtime.
The 1,856-byte tail of either .fconst is not an RMSNorm weight (measured on the 0.5B pack; the
ε claim holds, the norm claim does not — amd-oss/npu-re FACTS-oga-pack-weight-layout §7.1). That is a
separate open thread.

So this issue is the geometry generalisation: derive the .fconst layer-block stride and the two norm
offsets from the pack's meta N/K instead of the constants measured on one pack, then verify on the
second family with the same epsilon-anchor check (48/48 on Qwen2.5-0.5B today; 32 per bucket for
Llama). Not silence: the importer refuses these tensors by name today, and npu-pack info prints the
reason per tensor.

Blocked on nothing but the derivation — no device, no vendor runtime. The tile payload itself remains
blocked on #671 item 1's open fact (the .fconst weight payload is requantised, so it cannot be matched
against the ONNX tensors; see amd-oss/npu-re §7.5-§7.6).

Correction to the derivation sketched above, after checking it against the vendor class's own asserts
(amd-oss/npu-re docs/FACTS-oga-pack-weight-layout.md §7.6, corrected in 876d48b): a 34,816-byte tile
is 16 x 2,176-byte subvolumes — that much is certain — but it is not one mladfQuantMatrix
block. That class's constructor asserts subv_cols % NUM_ROWS == 0, i.e. kernel_N must be a multiple
of 128; at 16 subvolumes the only admissible shapes are (K=384, N=128) or (K=128, N=256), and neither
reproduces Qwen2.5-0.5B's observed 32 attention tiles (they give 48 and 70).

So the per-projection tile count is open, and this issue's headline holds only in the weaker form: the
.fconst layer stride and the two norm offsets are hidden-dependent because the attention payload grows
with N — the exact law that maps N to bytes is part of what has to be established.

That does not change what the issue asks for (derive the stride and both norm offsets from the pack's own
per-op N/K instead of the constants measured at hidden 896), but the derivation now has to be checked
against a second pack's byte layout rather than computed from the vendor headers alone. The honest
instrument is still the epsilon anchor, which gives 48/48 on the 0.5B pack today.

Triage (pi-issues): labelled enhancement, on the board. Owner: pi-ogaload — it is #671 item 4's second pack
family, and the derivation is host-side, so it belongs to the lane that owns the importer rather than to a new one.

What I will hold it to when it comes back: the same epsilon-anchor check that proved pack 1 — 48/48 on the
Qwen2.5-0.5B block layout today, and on amd/Llama-3.2-1B-Instruct_rai_1.8.0_npu_4K the stride and the two norm
offsets recomputed from cache/*_meta.json N/K (80 tiles, not 32) with the per-bucket anchors verified, plus the
145 refusals named in npu-pack info dropping to only the ones that are genuinely still open. It closes on that
check, not on "the file parses".

Not blocked on a device or the vendor runtime — noted, and that is why it can be done while the GPU is degraded
(#715) and the box is under the memory rule.

npu-pack: locate the .fconst layer blocks from their own anchors, for…

#734 — derived, measured on 1.7B and 4B, PR on 1bit/671-oga-pack-import @ b395928.

The stride and the two norm offsets are no longer constants for hidden 896. Every layer block opens with
the bf16 RMSNorm epsilon and carries a second such header right after the attention payload, so the
header positions repeat with one period per block; the stride is positions[i+P] - positions[i] for that
period (taken as the most common difference, so a header missed at a read-chunk boundary does not defeat
it — that is what failed on the 4B first time round).

Measured and verified on three architectures, 176 anchors:

pack	hidden	layers	stride	second header	anchors
Qwen2.5-0.5B-Instruct_rai_1.8.0_npu_4K	896	24	8,708,288	1,115,968	48/48
Qwen3-1.7B_rai_1.8.0_npu_4K	2048	28	27,931,456	4,739,136	56/56
Qwen3-4B_rai_1.8.0_npu_4K	2560	36	55,308,096	8,778,816	72/72

1bit npu-pack import now extracts 2 norms per layer at any hidden size (50 / 58 / 74 tensors per pack),
records the measured geometry in the container manifest, and tests/npu_pack_import derives its
expectations from the pack's own config, still asserting the hand-measured 0.5B offsets exactly when that
geometry is present.

One honest caveat on this issue's wording. The derivation was tried from the metadata first, and it
does not close the problem on its own. Per-projection N/K are declared (attrs.N / attrs.K), but the
attention payload's tile count does not follow N/64 x ceil(K/kernel_K): that predicts 96 tiles for
the 1.7B (measured 136) and 160 for the 4B (measured 252), because the writer's kernel shapes
come from a runtime supported-shapes table that the pack does not ship. So the offsets are established for
every hidden size by measurement today, and a closed-form stride from metadata alone remains open — the
table above is the evidence any such formula has to reproduce.

Related, and it changes the search space: the layer tile layout itself is now settled by a known-answer
write (feed chosen weights through bf16_const_params_shuffle in the shipped libdyn_dispatch_core.so,
read the tile back; 448/448 scales, 32/32 zero points, 4096/4096 codes). See
amd-oss/npu-re/docs/FACTS-oga-pack-weight-layout.md §7.13. What is still open there is not the layout but
which tile maps to which projection/row block, and whether the shipped weights relate to the upstream
checkpoint by the AWQ smoothing §2.2 recovered.

npu-pack: locate the .fconst layer blocks from their own anchors, for…

npu-pack: import a Ryzen AI OGA NPU pack into our container (#671 ite…

Not closing this on #780's merge yet — the delivery is real but the issue's own acceptance names a family the table does not include.

#780 (921179c1c, carrying b395928) has the derivation and it is honestly measured: 176 anchors over three geometries, stride and both norm offsets read from the file instead of the hidden-896 constants (896 → 8,708,288 / +1,115,968; 2048 → 27,931,456 / +4,739,136; 2560 → 55,308,096 / +8,778,816; 48/48, 56/56, 72/72), with the metadata-only closed form correctly reported as not available rather than faked.

What the acceptance in this issue asked for, and what my triage comment held it to, is amd/Llama-3.2-1B-Instruct_rai_1.8.0_npu_4K — the pack this issue was filed from, where the .fconst norms were 33 of the 145 refusals. One host-side command produces it:

1bit npu-pack info ~/models/oga/Llama-3.2-1B-Instruct_rai_1.8.0_npu_4K   # or: import -o /tmp/lla

and what would close this: 32/32 epsilon anchors on that pack (16 layers × 2 norms), or the refusal list dropping those 33 tensors to 0 with the measured geometry in the container manifest. Either that row, or a one-line statement that the three-geometry table (896/2048/2560, all Qwen) supersedes the Llama requirement — the 80/80 plan self-validation on Llama-3.2-1B is a different check (plan vs cache/*_meta.json), so it does not stand in for the anchors.

No device, no vendor runtime, no window: it is the same npu-pack info that filed this issue.

The one npu-pack info this row asked for: run. It names one predicate, not a geometry gap — and the derivation takes this pack unmodified.

Host-only, no device, no ticket: 1bit npu-pack info on ~/models/oga/Llama-3.2-1B-Instruct_rai_1.8.0_npu_4K, from a build whose npu/pack_import.cpp is byte-identical to main 801a4a60c (git diff --name-only 906f966a..origin/main → 0 hits for that file, and the build's own branch does not touch it).

195 tensors, 161 refused
  112  "layer blocks not located: fewer than 6 RMSNorm epsilon headers in the first GiB"
   48  MLP: 4,288-byte subvolumes … (the vendor packer emits only the 2,176-byte K=896 instance)
    1  final norm: the .fconst tail envelope is [eps][pad][448 f32] …

The 33 norm refusals in the filing are inside the 112 now, because the scan failure is attributed to every tensor in the block.

1. Root cause: the anchor is hard-coded to bf16 1e-6 and this pack declares 1e-5

npu/pack_import.cpp:246-251 eps_at() tests the two bytes 86 35; :282-290 is_eps_header() requires the same two bytes plus ten zero bytes; :306 scans for exactly that pair.
This pack's config.json says "rms_norm_eps": 1e-05 (the Llama default), i.e. bf16 0x3728 → bytes 28 37 — not 0x3586 / 86 35 (1e-6, the Qwen family's value: Qwen2.5-0.5B, Qwen3-1.7B and Qwen3-4B all declare 1e-06).
Measured over the whole 655 MiB file, not the first GiB: 0 hits for 86 35+10 zero bytes (36,409 loose 86 35 pairs — weight bytes), 33 hits for 28 37+10 zero bytes. The scan does not fail for want of geometry; it is looking for a constant this pack does not contain.
The pack's epsilon is already read elsewhere: npu/forward/src/q4nx_semantics.c:390 takes rms_norm_eps from config for the forward's arithmetic (npu/forward/include/common.h:39,62). Only the importer's locator is pinned at 1e-6, so the forward would compute with 1e-5 while the importer cannot find the block that holds the weight.

2. With the pack's own epsilon, #780's derivation takes it unmodified

33 headers; every position mod stride ∈ {0, 5,922,880}
P=2  d=33,640,640   31/31 pairs vote for it    -> stride 33,640,640 · h1 5,922,880 · period 2
pos[32] = 538,250,240 = 16 × 33,640,640        -> the 33rd header is the tail header, exactly at the end of the last block

So 16 blocks × 2 anchors = 32/32 epsilon anchors on Llama-3.2-1B plus the tail header. This issue's acceptance alternative 1 is one constant away, not a geometry away.

3. A correction to the row's premise: the closed-form table cannot be keyed on hidden size

#780's table has hidden 2048 → stride 27,931,456 (Qwen3-1.7B, intermediate 6144). This pack is hidden 2048 with intermediate 8192 and its stride is 33,640,640 — same q/k/v/o (2048/512/512/2048), same head_dim 64, same period 2. So the stride is a function of the whole geometry, MLP section included, and a three-row table is three samples rather than a rule. That is an argument for the derivation being the mechanism and the table being documentation — which is what #780 built.

4. For #844 while it is open: the tail envelope at hidden 2048

pos[32]+64 reads as 2,048 bf16 values: all positive, min 0.0417, max 2.922, mean 2.351 — shape-consistent with a final RMSNorm weight, and the 896 pack's [eps][pad][448 f32] is the same bytes read the other way (896 bf16 ↔ 448 f32), so hidden 2048 would be [eps][pad][1024 f32] / 2,048 bf16.

Not confirmed against a reference, and I am flagging that rather than banking it: neither that block nor layer 0's appears verbatim in model.onnx.data, searched as bf16 and as fp32. Related: layer 0's bytes at +64 are not a norm weight (mean 0.006, both signs — 4-bit payload), so this pack's in-block layout is not the 896 pack's. The anchors generalise; what sits immediately after them does not.

The row's next action is now one predicate

Read rms_norm_eps from the pack's config.json, convert to bf16 bits, and pass it into locate_fconst/is_eps_header (default 1e-6, so every existing pack stays byte-identical), plus the eps_at() verifier at :1153. If you would rather keep #842/#844 clean — both touch this file — say so and I will open it as its own one-file PR with a crafted-pack test (the pb writer in tests/npu_pack_import_test.cpp already builds synthetic packs; a fake fconst with seven 28 37 headers is enough to assert P=2 and the stride).

npu-pack: locate the fconst layer blocks with the pack's own epsilon,…

Closed by the merge of #887 (61b0d22332da0825fc8f36652de4019a962c1aa0). Verified at 65460184: the epsilon is threaded everywhere the locator needs it — npu/pack_import.h:66,179-187 (eps_bits, FconstGeom, bf16_bits_of, locate_fconst(path, eps_bits)) and npu/pack_import.cpp:246-248,276-302 (eps_at, is_eps_header, eps_header_positions all match the pack's own two bytes). The PR's own numbers are the acceptance this issue asked for: Llama-3.2-1B stride 33,640,640, second header 5,922,880, 32/32 anchors, 35 tensors / 32 layer norms imported — and the same hidden size (2048) with a different stride than Qwen3-1.7B, which is why the block has to be read out of the file rather than derived from a table.

Navigation Menu

Repository navigation

npu-pack: layer norms (and the final norm) are located only for hidden 896 #734

1. Root cause: the anchor is hard-coded to bf16 1e-6 and this pack declares 1e-5

2. With the pack's own epsilon, #780's derivation takes it unmodified

3. A correction to the row's premise: the closed-form table cannot be keyed on hidden size

4. For #844 while it is open: the tail envelope at hidden 2048

The row's next action is now one predicate

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu-pack: layer norms (and the final norm) are located only for hidden 896 #734

Description

Activity

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 1, 2026

bong-water-water-bong commented on Nov 2, 2026

1. Root cause: the anchor is hard-coded to bf16 1e-6 and this pack declares 1e-5

2. With the pack's own epsilon, #780's derivation takes it unmodified

3. A correction to the row's premise: the closed-form table cannot be keyed on hidden size

4. For #844 while it is open: the tail envelope at hidden 2048

The row's next action is now one predicate

bong-water-water-bong commented on Nov 3, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions