Repository navigation
npu: the Llama-3.2-1B pack imports whole and then drifts immediately — its first EOS is served as text #1044
Description
npu: the Llama-3.2-1B pack imports whole and then drifts immediately — its first EOS is served as text
Now that #844 lands, the Llama family is the second pack that converts whole (147 tensors, 150,848 q4_1 tiles,
1280/1280 attention tiles, 2560/2560 MLP groups, 32/32 norm anchors, model.norm.weight bit-identical to the
pack's own tail bytes). Its forward is not right yet, and this issue is the measurement rather than a guess.
Instrument: ~/ogaload/oga13/gate_llama.sh — the upstream meta-llama/Llama-3.2-1B-Instruct checkpoint in bf16 as
the reference, our container fed the reference's own prompt ids, 16 greedy steps, NPU_INFER_HOST_GEMM=1 (no
device, no ticket).
| prompt | ours | mean KL | median | first argmax split |
|---|---|---|---|---|
| capital of France | The capital of France is Paris.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nI'm sorry, |
2.49 | 0.00012 | step 12 |
| first three primes | The first three prime numbers are:\n\n1. 2\n2. 3 |
7.30 | 3.72 | step 6 |
| why the sky is blue | The sky appears blue because it is primarily composed of a mixture of gases, including |
8.29 | 10.10 | step 5 |
Two different defects are visible in that table, and they should not be conflated:
- A stop-token defect that is ours, not the pack's. The capital prompt is token-for-token right through the
answer (The capital of France is Paris.) and the next model token is<|eot_id|>— after which the reference
stops and we keep generating, emitting the control token as text and inventing a second turn. That is our
serving path not treating this family's EOS as a stop (Llama-3.2 uses 128009<|eot_id|>and 128001
<|end_of_text|>), and it is easy to confuse with a model defect because the text looks broken. - A genuine drift from a correct prefix on the other two prompts (5 and 6 steps, KL ~7-8 at the split), which
is a weights/layout question like engine#1003 was for the Qwen3 packs — not a near-tie: the KL at the split is an
order of magnitude above the 0.03-0.4 the Qwen packs show at their coin flips.
What is established, so the next step starts from evidence: the container reads the pack (147 tensors, 16 layers,
H=2048 NL=16 NH=32 NKV=8 HD=64 IM=8192 NV=128256, qk_norm=0, rope_theta=500000 from the pack's own config),
the import re-encodes every tile and group to the pack's bytes, the final norm is bit-identical to the pack's tail,
and the four array designs are named by the forward (QKV K=2048 N=3072, O K=2048 N=2048, GU K=2048 N=16384,
D K=8192 N=2048; the b8 ELFs for them exist in ~/ogaload/oga13/kernels-llama-b8).
What is not established, and is the experiment that decides it: whether a Llama block's projections in the
container are the checkpoint's tensors under the layout the importer assumes (the check §7.25 ran for Qwen3 layer 0-13
— dequantise a container tensor, correlate against the checkpoint after the per-input-channel fold). Llama's section
geometry differs from Qwen2.5 in at least the ways already recorded in FACTS §7.24 (the k/q/v run order is k, q, v)
and its eps is 1e-5 rather than 1e-6, so "it worked for Qwen" is not an argument. Candidates to exclude in order:
attention payload split, MLP gate/up order, and the o_proj K geometry (o_proj_k is recorded as 2048 for this pack,
i.e. K == hidden, which the 4B Qwen3 pack does not satisfy).
Activity
bong-water-water-bong commentedon Nov 5, 2026
Answering this issue's "not established" question, and correcting one of its readings. All device-free
(
NPU_INFER_HOST_GEMM=1, no NPU ticket, nothing on:18097).1. The container does hold the checkpoint's tensors
The experiment the issue asks for, run on Llama (not carried over from Qwen): dequantise a container tensor with
the q4_1 tile rule, correlate against
meta-llama/Llama-3.2-1B-Instructbf16.corrFfits out the oneper-input-channel scale the packer folds into a projection.
corr.T ≈ 0everywhere rules out a transposed read.lm_headdequant vs the tied checkpoint embedding: corr +0.9942.model.norm.weight: 2048/2048 bit-exact,while the per-layer norms match 0-22 of 2048 channels — exactly the channels whose AWQ smoothing factor is 1, which
is the fold, seen from the other side.
Two tensors read low on corrF (L15
v_proj+0.9479, L15up_proj+0.9693) and they are not defects:per-output-row correlation is +0.979 min / +0.993 median for v_proj (0 of 512 rows below 0.8) and +0.976 / +0.994 for
up_proj (0 of 8192). The packer applies a second, per-output-row fold to the tensor that precedes an AWQ scaling
op — the container's own
algo_configsayso_proj.prev_op = self_attn.v_projanddown_proj.prev_op = mlp.up_proj— and a per-column-only fit cannot recover that. So: the attention/MLP split, the gate/up order and o_proj's K
geometry named in the issue are all excluded.
2. The "drift" is one coin flip per prompt, not drift
The greedy table above can't say that, because after the first split the two sides generate from different token
histories — every later row compares different contexts, and that is where the KL 7-8/worst-case rows come from
(the per-position KL at the split is 1.115 / 0.978 / 1.385, not 7-8).
Re-measured on identical context: serve once, then for each position
ksendprompt = ref_prompt_ids + ref_out_ids[:k]withmax_tokens 1and keep the dumped logits. Every row now shares the reference's prefix, androws below the split reproduce the greedy numbers to the digit (instrument control).
6070421.50 vs ours4019.79 (ref's own #2 is4019.625)147326.21 vs ref22024.1331519.38 vs ours43317.37 (we rank315second at 16.12)Agreement resumes right after each split (p2 rows 7-15: KL ≤0.0006, top-1 on all nine). So the forward is
per-position correct and flips once per prompt, on a different position each time, among the same handful of
candidates — the Qwen3 near-tie signature, not a layout error.
3. A separate defect this family exposes:
rope_type: "llama3"is not implementedThe pack declares
rope_scaling {llama3, factor 32, high_freq_factor 4, low_freq_factor 1, original_max_position 8192}withrope_theta 500000; the importer carries it into the container verbatim, and theforward builds RoPE as
theta^(-i/half)and nothing else (origin/mainnpu/forward/src/q4nx_forward.cpphandlesLongRoPE
short_factorandfreq_factors, no llama3 branch;q4nx_semantics.hhas no scaling fields).Measured against transformers as the control, my port of the formula reproduces its
inv_freqto 4.2e-8:Not the cause of §2's flips (0.01 rad at position 5-12 cannot move a logit by 1.9-3.2), but it grows with context and
will dominate at this pack's real 4096-32768 range. Filed separately rather than folded in here.
4. Defect 1 (EOS served as text) — root cause found, fix in flight
app/forward_serve.cppreadeos_token_idwithjson_int(), which answers its default for anything that is not aninteger, so this pack's list
[128001, 128008, 128009]gaveeos = -1and the serve's only stop id was whatever<|endoftext|>resolved to by name. Reproducer (device-free):finish_reason: length, textThe capital of France is Paris.<|eot_id|><|start_header_id|>assistant<|end_header_id|>\n\nI'm sorry,.npu/model.cppandnpu/pack_import.cppalready parse the array form — only the serve path was scalar. PR to follow.bong-water-water-bong commentedon Nov 5, 2026
Defect 1 fixed in #1052 (
f50f600). Root cause was one line:app/forward_serve.cppreadeos_token_idwithjson_int(), which answers its default for a list, so this pack's[128001, 128008, 128009]gave no stop ids and<|eot_id|>128009 was served as text. Same reproducer, rebuilt:Defect 2 stays open and is measured in the comment above (15 of 16 positions per prompt agree on identical context, one flip each); it is not an eos problem. The
rope_type: llama3gap found while measuring it is #1048.forward-serve: read eos_token_id as a list, so a Llama container stop…
bong-water-water-bong commentedon Nov 5, 2026
Defect 2 is attributed, and it is not our forward
§7.30 above answered how many positions disagree (15 of 16 per prompt on identical context, one flip each). This
answers why, with an experiment that separates the only two candidates — the pack's 4-bit weights, or our forward —
which the checkpoint comparison cannot do, because both are present in it.
Method (device-free, ~10 min of CPU, no AMD software): load the upstream checkpoint in fp32 in transformers,
swap in the container's own dequantised tensors (146: 112
q4_1projections, 34 bf16 norms/embedding, plus thepack's separately quantised
lm_head; the vectorisedq4_1dequant was checked bitwise against the reference loopfirst), then step it greedily on the same prompt ids. That is a reference implementation of the pack's model: it
shares our data and none of our code.
60704404022014731473315433433Every flip reproduced, at the same position, with the same token. And our engine against that reference
implementation: 48 rows, zero argmax splits, corr 0.999-1.000, KL mean 0.00085 / 0.00005 / 0.00035 (max 0.00938 /
0.00075 / 0.00203) — the residual is fp32 reduction order.
So: the flips are entirely the pack's 4-bit weight quantisation. The container reads the pack correctly (#1052,
§7.28), the forward executes the pack's model correctly, and what remains is the distance between a 4-bit pack and the
bf16 checkpoint it was made from — which any correct engine would show.
That also means the framing in this issue's body needs one correction: a single comparison against the upstream
checkpoint measures two different things at once, and they should be reported separately — engine fidelity (our
logits vs a reference implementation on the pack's weights: KL ~1e-4, no ranking changes) and the pack's own
distance from its upstream checkpoint (KL mean 0.093 / 0.052 / 0.036 plus one flip per prompt). The second number is
not a defect in anything; it is the price of 4-bit.
Recorded as
npu-reFACTS §7.31 (5037b09, branchogaload/oga-pack-layout-20261031), evidenceevidence/oga/llama-family-2026-11-04/attr_flips.py.