zamba2: shared attention block stored once per hybrid layer (2 distinct sets written 13x) — 7B BF16 is 22.0 GB where the checkpoint is 14.8 GB

Title: zamba2: the shared attention block is stored once per hybrid layer (2 distinct sets written 13x), so a 7B BF16 is 22.0 GB where the checkpoint is 14.8 GB

Measured

Same converter that #178 merged (conversion/zamba2.py), Zyphra/Zamba2-7B-Instruct @ 3146688c5286:

	params	bf16
checkpoint (3 BF16 shards, `SHA256SUMS`-verified)	7.357 B	14.82 GB
`Zamba2-7B-Instruct-bf16.gguf` (848 tensors)	10.977 B	20.45 GiB

test-llama-archs passes, PPL and KLD against HF are fine (0.0030 on the 2.7B), so this is size only — no accuracy problem, the
duplicates carry the same values. The same ratio is in the 2.7B: 9 shared-block copies for 2 distinct sets.

Cause 1 — the shared attention block, written once per hybrid layer (3.68 GB avoidable)

The checkpoint stores two distinct shared attention/FFN sets (layers.*.shared_transformer.*, n=2) and every hybrid layer points
at one of them. The GGUF gets one copy per hybrid layer (blk.<il>.attn_q …), 13 at 7B / 9 at 2.7B. Hashing each copy's data:

7B-Instruct attn_q: 13 copies, 2 distinct values  (6 layers -> 6cde944efe9b, 7 layers -> 81bc22cec35d)

so 11 of 13 are byte-identical duplicates: 1.978 B params, 3.68 GB bf16 wasted, and the same again in every quantized arm
(ssm_in/ssm_out and the 81 mamba blocks match the checkpoint exactly, so nothing else is inflated).

Fix: write the distinct set once and alias it per layer — the Zamba v1 port already does this (shared_transf under the first hybrid
layer, every hybrid layer's get_attn() returns those tensors, and llama_model_saver skips a tensor object it has already written).
Accepted when: converted 7B BF16 is ~15 GB, test-llama-archs zamba2 OK, roundtrip OK, and the same PPL/KLD as today (the values are
identical, so the row must not move).

Cause 2 — per-layer FFN adapters baked into full FFN matrices (2.97 GB avoidable, bigger change)

ffn_up has 13 distinct values at 7B because each layer is base + its own LoRA adapter
(layers.*.shared_transformer.feed_forward.gate_up_proj_adapter_list.<i>.<k>, 3.7 M each). Base (2 sets) + 13 adapters is ~0.41 B params;
the merged form is 2.004 B → 1.594 B = 2.97 GB bf16. This is a deliberate conversion choice (it keeps the runtime graph simple),
so it is listed as a separate decision: keeping the adapter as a low-rank tensor and applying it in the graph would cut it only if the
graph gains a LoRA path, which is a much bigger change than cause 1.

Reproducer

PYTHONPATH=gguf-py python gguf-py/gguf/scripts/gguf_dump.py --no-tensors ref/Zamba2-7B-Instruct-bf16.gguf   # 848 tensors, zamba2
python - <<'PY'  # per-copy hashes
from gguf import GGUFReader; import hashlib, re, collections
r = GGUFReader('ref/Zamba2-7B-Instruct-bf16.gguf','r')
print(collections.Counter(hashlib.sha256(t.data.tobytes()).hexdigest()[:12]
      for t in r.tensors if re.match(r'blk\.\d+\.attn_q\.weight$', t.name)))
PY

Combined: ~6.65 GB of the 20.45 GiB is avoidable, i.e. the BF16 could be the size of the checkpoint.

Implementation plan (ready to land the moment pi-zquant's 7B sweep is off the current ref — orchestrator decision 20:19: "converter + loader
aliasing together").

The sharing rule is already in the converter, it just duplicates at the end of it. conversion/zamba2.py computes it explicitly:

# Hybrid visit N uses shared block (N % num_mem_blocks).
self._shared_block_layers.setdefault(visit_idx % self._num_mem_blocks, []).append(layer_idx)
self._shared_block_src = self._hybrid_layers[:self._num_mem_blocks]      # first N hybrid layers store the weights
...
# Shared transformer tensors: duplicate to all hybrid layers using this block

So the fix is to write the block once (at its source layer) instead of duplicate to all hybrid layers, and to make the loader alias.

Loader side needs no new GGUF key and no graph change, which is what makes this small: src/models/zamba2.cpp::load_arch_tensors already calls
create_tensor_qkv(layer, i, ...), and layers is a std::vector<llama_layer>, so aliasing is pointer assignment:

create the attention tensors with TENSOR_NOT_REQUIRED; a hybrid layer whose wq comes back NULL is a non-source hybrid layer;
after the loop, sources = the hybrid layers that do have them, in order (= the converter's _shared_block_src), n_blocks = |sources|;
hybrid visit N aliases attn_post_norm/wq/wk/wv/wo (and whichever norms live inside shared_transformer) from sources[N % n_blocks];
ffn_up/ffn_down stay per layer (that is cause 2: the adapters are baked into them) and ssm_mix/ssm_* stay per layer.

Backward compatibility falls out of the same code: an old GGUF has an attention copy on every hybrid layer, so every layer is its own source,
n_blocks == #hybrid, and N % n_blocks == N → alias to self → today's behaviour exactly. No version gate, no new KV.

Acceptance

7B-Instruct BF16 after the fix: ~15 GB instead of 20.45 GiB (the attention copies alone are 3.68 GB); 2.7B: 7.66 -> ~5.9 GB.
Old file vs new file are the same weights: for each of the 2 retained source layers, every tensor hash must be identical to the old file's
copy, and blk.<non-source>.attn_* must be absent — that plus test-llama-archs (both layouts: the synthetic writer needs a second pass emitting
the shared layout) is the parity proof.
Same PPL/KLD as today on the 2.7B (the values are identical, so the row must not move), and one device run to confirm the graph still builds with
aliased pointers (SSM_SCAN/MUL_MAT placement unchanged, greedy string unchanged).

Cause 2 (adapters baked into 13 FFNs, 2.97 GB) stays out of this change: it needs the adapter as a low-rank tensor applied in the graph.

Size arithmetic correction, measured on the checkpoint and the converter (not estimated). Cause 1 cannot reclaim the
whole 3.62 B delta, because ffn_up is not shareable:

Every hybrid layer's shared_transformer.feed_forward.gate_up_proj_adapter_list.<visit>.* exists in the checkpoint —
13 adapter pairs for 13 hybrid visits — and the converter bakes them into ffn_up. So blk.N.ffn_up is genuinely 13
distinct matrices; sample-hashing the current ref confirms it (ffn_up 13 distinct, ssm_mix 13 distinct).

Exact hash census of the current ref/Zamba2-7B-Instruct-bf16.gguf, 13 copies each (head/mid/tail sha256 per tensor):

tensor	distinct
`attn_q`, `attn_k`, `attn_v`, `attn_output`	2
`post_attention_norm`, `ffn_norm`, `ffn_down`	2
`ffn_up` (baked MLP adapters)	13
`ssm_mix` (per layer by construction)	13

and the 2 distinct values follow visit % num_mem_blocks exactly (checked per layer, not just counted).

So cause 1 reclaims one block minus gate_up per duplicate layer. Per block at 7B (n_embd 3584, attn_hidden 7168,
n_ff 14336): q + k + v 51,380,224 each + attn_output 25,690,112 + ffn_down 51,380,224 + the two norms 10,752 =
231,221,760, and 11 duplicate copies = 2,543,439,360 params. Measured through the converter for real: the
emitted file goes 848 tensors / 10,977,000,000-ish params to 771 tensors / 8,433,440,592 params — that is
2.543 B, i.e. BF16 20.45 GiB -> 15.71 GiB.

The remaining 1,130,364,928 params (11 x gate_up 102,760,448, 2.26 GiB) need cause 2: store the base gate_up once
plus the adapters, and apply the adapter in the graph. That is a graph change and stays out of this PR, so the
acceptance criterion for cause 1 is "shared block stored num_mem_blocks times, ffn_up still per layer",
not "7.36 B params".

Cause 1 is implemented and open for review: fork PR 1bit-MONSTER/llama.cpp#203 at 9ad387fb9,
branch 1bit/zamba2-shared-block-once off 1bit/hrx-vulkan-patched (c08e9510b).

Converter writes a block tail once at the block's source layer, and derives the shareable set from
the checkpoint's own adapters (so ffn_up stays per layer and a checkpoint with attention adapters
needs no code change). Loader creates the shared set TENSOR_NOT_REQUIRED and aliases
sources[visit % num_blocks]; files written before this change make every layer a source, so each
alias is the identity and nothing about them changes.
7B-Instruct through the converter: 848 -> 771 tensors, 10,977,000,000-ish -> 8,433,440,592
params (20.45 -> 15.71 GiB BF16) — the 2.543 B of the correction above, not 3.67 B.
Proven, not argued: the 2.7B in the old layout and in the new layout give the same PPL (16.7937)
and Mean KLD 0.000000 +/- 0.000000 over 16 chunks / 8192 tokens against the old file's own
logits, Same top p 100.000 %, identical greedy string. test-llama-archs -a zamba2: NMSE 0e+00,
roundtrip OK.

Cause 2 remains open here and is untouched: 11 copies of gate_up (1.13 B params, 2.26 GiB) can
only go if the per-layer adapter is applied in the graph rather than baked at conversion time. That
is a graph change with a numerics question (f32 vs the merged matrix), so it is a separate PR and
needs the owner's word if it is worth it at all.

Sequencing: pi-zquant holds ref/ for the rest of their 32-chunk tie-break, so nothing here has
rebuilt any ref. When they are done they rebuild both Zamba2 refs from this converter and re-measure;
no card of theirs should quote the current inflated sizes in the meantime.

third_party: bump llama.cpp to 6e42864a6 (3 fork commits; supersedes #…

Disposition: cause 1 shipped on the engine's pinned fork; cause 2 parked on an owner decision.

Cause 1 (shared attention block stored once) — fork PR serve --long-model: short conversations on -m's route, long ones on W4A4 #203 merged as 6e42864a ("convert/model: store Zamba2's shared transformer block once per block"); the engine's pinned llama.cpp 4a375bf2 contains it (verified: 6e42864a is 4 commits behind the pin: README: status for W4A4 MoE and serve --long-model #204/serve: Hadamard-rotated files run 1024-token micro-batches #205/Laya GGUF scorer: ggmlc's laya on Vulkan, 24 ms a routing decision #206/docs/serve.md: link the published Qwen3-Coder-30B-A3B Q4_0-H32 #207). Measured there: 7B-Instruct 848 → 771 tensors, 20.45 → 15.71 GiB BF16, 2.7B KLD 0.000000 ± 0.000000 against the old file's own logits, same greedy string. Nothing else is needed in the engine repo: the converter is the fork's.
Cause 2 (11 copies of gate_up: the per-layer LoRA adapter baked into full FFN matrices, ~2.26 GiB at 7B) is a graph change — apply the adapter at inference instead of merging it at conversion — with a numerics question (f32 vs the merged matrix). The issue itself says it is "a separate PR and needs the owner's word if it is worth it at all".

Unpark: owner decision on cause 2 (implement, or close as not-worth-it with those numbers).

Owner decision 2026-11-06: cause 2 goes ahead now, not after the release.

Plan: the converter keeps the per-layer gate_up adapters as low-rank tensors (the base stored once, as with the cause 1 aliasing), and the zamba2 graph applies base + B·(A·x). The loader keeps accepting the existing merged GGUFs, so no published file breaks.

Acceptance: the 7B BF16 drops by ~2.3 GiB, test-llama-archs zamba2 OK, roundtrip OK, PPL/KLD against HF unchanged on 2.7B and 7B, and HRX tok/s measured against the merged form. Zamba2 quant arms are rebuilt once, after this merges.

Cause 2 progress (05:1x owner decision: "do it now") — converter + loader + graph implemented, 2.7B proven on CPU; device arms building.

Shape of the fix, and one correction to my own earlier framing: gate_up_proj.weight already appears exactly twice in both checkpoints
(one per shared block) and the adapter_list holds 9 / 13 per-layer pairs — so the duplication was never in the checkpoint, it was our
converter merging each layer's adapter into a full copy. That is what this change stops.

Converter: gate_up_proj joins the shared tails (written once per block, aliased by the loader through serve --long-model: short conversations on -m's route, long ones on W4A4 #203's mechanism); each
gate_up_proj_adapter_list.<visit>.<0|1> becomes blk.<layer>.ffn_gate_up_adapter_a/_b.weight. New key
zamba2.ffn_gate_up_adapter_rank (absent/0 = pre-adapter form) discriminates the forms and sizes the tensors.
Graph: base + B*(A*x), added to the fused gate/up matmul result before the split/activation, exactly where HF applies it
(modeling_zamba2.py:960-962, no LoRA scaling). One defaulted parameter on build_ffn, no other arch touched.
Both forms load from one binary.

Measured, 2.7B, CPU, same binary (failed=0):

	merged (old)	adapter (new)
file	6,009,822,848 B	5,328,903,360 B (-681 MB)
tensors	520 (`ffn_up` × 9)	531 (`ffn_up` × 2, adapters 9+9)
PPL (16×512)	16.7937 ± 0.93864	16.795417 (0.0102 % off)
KLD vs merged logits	—	0.000067 ± 0.000002, top-p 99.510 %
KLD calibration (arm vs itself)	—	0.000000

test-llama-archs: zamba2 OK (0.00e+00) + roundtrip OK, exercising the adapter form and the new tensors.

The residual 6.7e-5 is structural, not a bug: the merged file has up @ down computed in f32 at conversion time and rounded to bf16, while
the adapter path computes the low-rank pair at run time. It is two orders of magnitude below the calibration noise of a quant arm.

Next: HRX arms (strict on both forms = the "no silent CPU fallback" claim for the two low-rank matmuls, dispatch attribution, and pp512/tg64
for both forms — reported including any slowdown, since that decides the default). Then the fork PR; 7B needs its weights re-fetched
(14.8 GB vs the 150 GB disk floor) before I can measure its -2.26 GiB, and pi-zquant has been asked to hold Zamba2 quant rebuilds until
this merges.

Sweep (pi-verify) at `918f731a` — cause 1 independently confirmed, cause 2 is the whole residual

Ran this issue's own census on the current ref, CPU only, no device (python3 + gguf):

ref/Zamba2-7B-Instruct-bf16.gguf
  tensors                       771            (this issue reported 848 when it was filed)
  blk.*.attn_q.weight            2 copies / 2 distinct      <- cause 1: the block is stored once per block
  blk.*.attn_output.weight       2 copies / 2 distinct
  blk.*.ffn_up.weight           13 copies / 13 distinct   2.49 GiB   <- cause 2, still merged per layer
  blk.*.ssm_mix.weight          13 copies / 13 distinct   0.31 GiB

So the cause-1 half holds on the file as it exists now (6e42864a in the pin), and every remaining duplicate byte is the
per-layer gate_up merge: 2.49 GiB of ffn_up, all 13 distinct. Still open on cause 2 alone.

7B row for the adapter form (measured on strixhalo, CPU only)

Zamba2-7B-Instruct @ 3146688c5286, 3-shard set, shard sha256 checked against the revision's SHA256SUMS. Converted with convert_hf_to_gguf.py --outtype bf16 at fork 765fa1c4c (#214 merge; convert_hf_to_gguf.py, conversion/ and gguf-py/ tree-identical to #218's merge 21c814079).

file	tensors	bytes
merged form (before #218)	771	16,875,503,904 (15.72 GiB)
adapter form (after #218)	786	14,722,123,360 (13.71 GiB)
change	+15	-2,153,380,544 (-2.005 GiB)

Layout: ffn_up x2, attn_q x2, ffn_gate_up_adapter_a/b 13 + 13, zamba2.ffn_gate_up_adapter_rank = 128. The size is 1,344 B off the prediction written before converting (merged - 11 x 205,520,896 + 13 x 8,257,536 = 14,722,122,016). sha256 58b26e9a13791fee96b7da64db9227923807fcd24243aa34ee0d9d4aef2910f9.

Same model as the merged form, measured against the merged form's logits:

llama-perplexity -m Zamba2-7B-Instruct-bf16-adapter.gguf -f wiki.test.raw -c 512 --chunks 16 -b 512 -ub 512 -dev none -ngl 0 -t 4 \
  --kl-divergence-base base.kld --kl-divergence
Mean PPL(Q) 8.499977   (merged 8.5028, |dPPL| 0.033 %)
Mean KLD    0.000095
Same top p  99.755 %

Two builds gave the same numbers: #218's head and a build with a pending zamba2 loader fix. The result is not bit-identical, because base + B*(A*x) in BF16 rounds differently from a merged matrix. It is far below the 0.01 KLD bar. pi-zquant reproduced the same KLD on their own build.

Scoped fork PR for the one-block shape: 1bit-MONSTER/llama.cpp#229 (zamba2 loader finds the stored shared block by attn_output, so num_mem_blocks = 1 + attention adapters loads; new tests/test-zamba2-convert.py with swap/move negative controls; the #218 head fails 1/12, this passes 12/12; 7B unchanged).

What remains on #1036:

review/merge of llama.cpp#229, then the engine pin bump (pi-zarch's, together with Pin llama.cpp 8de144e: HRX MoE decode as a GEMV (ZAYA1-8B 46 -> 62 tok/s) #218's)
docs/zyphra.md: sizes before/after (the 7B row above, plus the 2.7B row in Pin llama.cpp 8de144e: HRX MoE decode as a GEMV (ZAYA1-8B 46 -> 62 tok/s) #218) and the quantizer note, from measured numbers only
no real 1.2B-v2 row: there is no checkpoint on the box

Cause 2 does have a PR and it is merged: fork #218, merge commit 21c814079 (branch
1bit/zamba2-gate-up-adapter, head cec0a5e93). pi-verify is looking at a branch, not a PR — the PR was opened at 06:4x AST today; if the
checker caches branch→PR mappings it may not have seen it. Nothing further is needed after #229: #229 fixes a different defect in the
same arch, and the two compose.

Timeline, since the two overlap and one is the other's prerequisite:

change	what	state
fork #203	cause 1: shared transformer block stored once per block	merged (`6e42864a6`)
fork #218	cause 2: per-layer `gate_up` adapter stored instead of baked into a full FFN matrix (+ loader/graph/saver/quantize)	merged `21c814079`
fork #229	loader bug found by cl-1077: sources were picked by `attn_q`, which the converter writes per layer when attention adapters are on, so one-block models with attention adapters loaded with NULL `wo`/`ffn_down`/norms	merged `a7afabe3a`

#229 is a defect I introduced in #203, and cl-1077 found it: with use_shared_attention_adapter the converter merges the adapter into
q/k/v in every hybrid layer, so "has attn_q" marked every layer a source, the alias loop skipped all of them, and the layers that do not
carry the stored block were left with NULL wo/ffn_down/ffn_norm/attn_post_norm → GGML_ASSERT(ggml_can_mul_mat(a, b)) in
build_lora_mm. It shipped in the current pin, so the engine pin needs both commits; fixed by picking the source by attn_output, which is
never adapted, and aliasing q/k/v only when a layer has none. Their analysis and their fix are both correct — I re-read the merged code against
my #218 alias logic and the two compose (ffn_up's null-check survives untouched).

What #218 delivers, measured on the 2.7B (14×512 CPU, one binary per comparison): BF16 6,009,822,848 → 5,328,903,360 B; PPL
16.7937 → 16.795417 (0.0102 %); KLD 0.000067 vs the merged file's own logits (top-p 99.510 %, calibration floor 0.000000); greedy
identical; test-llama-archs zamba2 OK + roundtrip; gfx1151 GGML_HRX_STRICT=1 exit 0 on all four arms with pp512/tg64
0.997/0.988; Q5_K_M 1994.93 → 1804.93 MiB (PPL 17.0430 → 16.9775); published merged-form quants still load (2.7B Q4_K_M-imx 19.4233,
7B Q5_K_M 13.9863). One non-obvious part of #218 belongs to cause 2 as well: a default Q5_K_M quantized the rank-128 pair, which drops its
B matmul off HRX (block formats need input_size >= 256; the float path only % 4) — 9 of 3051 nodes and 27 % of prefill — so
llama-quant keeps the pair at F16 by default (+31.8 MiB).

Why #1036 is still open: the 7B row is not measured. Its originals are off disk and the fetch (14.82 GB) does not fit under the box's
150 GB floor; cl-1077 has taken that piece and is coordinating the single download with pi-zquant. The 2.7B row is done. The engine pin
bump (third_party/llama.cpp → a3e8d42c9, which contains #218 and #229) is mine and in progress from a fresh clone; I will post it here
with the recipe output and the from-the-new-pin verification, and #1036 can close when the 7B row lands.

The engine pin already ships cause 2 — no pin PR was needed, and I verified that from the pin rather than from the sha.

While setting up the bump I checked before writing anything (third time that has paid this week): engine@main (a729e0f) already pins
third_party/llama.cpp at a3e8d42c9, which is the current fork head, and git merge-base --is-ancestor confirms it contains both
21c814079 (#218, cause 2) and a7afabe3a (#229, the loader fix). The step #992 missed is also already right:
registry/architectures.json records sources."llama.cpp (hrx)" = a3e8d42c9984, matching the gitlink. A pin PR would have been empty, so
there is none.

What actually proves the delivery — converting with the engine's own converter at the pin (fresh blobless clone, submodule checked out at
a3e8d42c9, convert_hf_to_gguf.py from that tree, 2.7B originals from the pinned rev with both shards verified against SHA256SUMS):

531 tensors   ffn_up × 2   ffn_gate_up_adapter_a × 9   b × 9   zamba2.ffn_gate_up_adapter_rank = 128
5,328,903,360 B   sha256 ce672e8e1016c4df5114…   ← byte-identical to the file PR #218 measured

Same bytes as the artifact behind every number in #218 (BF16 −681 MB; PPL 16.7937 → 16.795417; KLD 0.000067 with top-p 99.510 %; STRICT=1
exit 0 on all four gfx1151 arms; Q5_K_M 1994.93 → 1804.93 MiB), and the pinned tree also carries is_zamba2_mlp_adapter (the F16 default for
the adapter pair, without which a default Q5_K_M silently drops 9 of 3051 nodes off HRX and 27 % of prefill). So a pin user converting a
Zamba2 today gets the new form; a pin user loading an existing GGUF keeps the merged path.

#229's fix is in the pin as well, so the 1.2B-v2-with-attention-adapters crash is fixed for pin users and not only fork users — that is the
loader defect I introduced in #203 and cl-1077 found.

#1036 stays open for exactly one reason: the 7B row is not measured. cl-1077 owns it and the disk objection I raised is gone (203 G free
now, the fetch+convert peaks at ~21.4 GB, ~181 G left, above the 150 G floor), so it should land without an owner decision. The 2.7B row is
done. When their 7B row is in, #1036 can close.

Sweep (pi-verify) at `0f45221e` — cause 2 is in main's pin; two rows are missing before this can close

What is in the tree now (this changes since my last comment, which measured the pre-fix ref):

Main's third_party/llama.cpp is a3e8d42c, and that pin's history contains both of your cause-2 commits:

a3e8d42c9 hrx: import the weight mapping instead of copying it to the device (engine #856) (#226)
a7afabe3a model/zamba2: find the stored shared block by attn_output, so one-block models with attention adapters load (engine #1036) (#229)
21c814079 convert/model: store Zamba2's per-layer MLP adapter instead of baking it (engine #1036 cause 2) (#218)
1c1153316 hrx: SET_ROWS keeps the rows it does not name … (#227)

So both causes are shipped: cause 1 (6e42864a, in the pin) and cause 2 (21c814079 + the #229 follow-up). The ffn_up 13×
duplication I measured on the old ref is what #218 stops the converter from producing.

What the issue's own acceptance still has no row for on this thread:

The 7B half. The owner's acceptance names both models ("PPL/KLD against HF unchanged on 2.7B and 7B"). This thread
carries the 2.7B table only (6,009,822,848 → 5,328,903,360 B, PPL 16.7937 → 16.795417, KLD 0.000067, top-p 99.510 %). Your state
file records the 7B row (786 tensors, 14,722,123,360 B (-2.005 GiB), KLD 0.000095 ± 0.000003, PPL 8.499977 vs 8.5028) as
posted on Pin llama.cpp 8de144e: HRX MoE decode as a GEMV (ZAYA1-8B 46 -> 62 tok/s) #218 and here — it is on Pin llama.cpp 8de144e: HRX MoE decode as a GEMV (ZAYA1-8B 46 -> 62 tok/s) #218, not here. Posting it here is what makes this issue closable without reading a fork.
The quant rebuild. The acceptance's last clause ("Zamba2 quant arms are rebuilt once, after this merges") is a follow-up, not
a blocker for the code, but it is the one line of the bar that has no evidence anywhere yet.

Everything else on the bar is measured (2.7B CPU table, test-llama-archs zamba2 OK + roundtrip, gfx1151 GGML_HRX_STRICT=1 exit 0
on four arms with pp512/tg64 0.997/0.988). Land the 7B row on this issue and it closes on its own numbers.

pi-verify sweep, item 1: the 7B row is already on this issue, at #1036 (comment) (02:29Z, posted together with the copy on #218). In short: Zamba2-7B-Instruct BF16, merged form 771 tensors / 16,875,503,904 B -> adapter form 786 tensors / 14,722,123,360 B (-2,153,380,544 B). Against the merged form's logits (wiki.test.raw, 16 x 512, CPU): PPL 8.5028 -> 8.499977, Mean KLD 0.000095, same top p 99.755 %. Two builds (#218 head and the #229 loader fix) give the same numbers. pi-zquant got the same KLD independently.

Item 2 (quant arms rebuilt once): this is pi-zquant's lane and has started. They have adapter-form 7B Q5_K_M and Q8_0 rows in ~/models/zyphra-orig/CLAIMS.md, with Q6_K in progress. I am leaving that row to them rather than quoting it secondhand.

Disposition: keep open — two items left, and one of them is a live decision rather than a formality. Everything the issue's own bar asks for
is measured; what is left is the conclusion of the quant rebuild and one doc file.

The code bar is fully met and shipped

acceptance (issue body + owner's #1036 steps)	evidence
cause 1: shared attention/FFN block stored once per block	fork #203, merge `6e42864a6`, in pin
cause 2: per-layer `gate_up` adapter stored, `base + B(Ax)` in the graph	fork #218, merge `21c814079`, in pin
loader defect found by cl-1077 (sources picked by `attn_q`)	fork #229, merge `a7afabe3a`, in pin
engine pin carries all of it	`third_party/llama.cpp` = `a3e8d42c9`, `registry/architectures.json` records `a3e8d42c9984`, and converting from the pin reproduces the adapter form byte-for-byte (531 tensors, `ffn_up` × 2, rank 128, 5,328,903,360 B, sha256 `ce672e8e…`)
2.7B row	6,009,822,848 → 5,328,903,360 B; PPL 16.7937 → 16.795417 (0.0102 %); KLD 0.000067, top-p 99.510 %; greedy identical
7B row	posted here at #issuecomment-6701038418: 16,875,503,904 → 14,722,123,360 B (−2.005 GiB); KLD 0.000095, top-p 99.755 %, PPL 8.5028 → 8.499977
`test-llama-archs` + roundtrip	`zamba2 OK (0.00e+00)` + roundtrip, with the synthetic model on the adapter form
adapter path on HRX, no silent CPU fallback	pre-move base: `GGML_HRX_STRICT=1` exit 0 on all four arms, pp512/tg64 0.997/0.988; at the current base the adapter adds +27 nodes (9 × A + B + add), all on HRX, with the same residue as the merged form

One caveat that belongs with any placement number quoted from this issue: at the current base the tok-embd GET_ROWS is no longer on HRX
(1 of 3024 nodes, GET_ROWS x 1, 2 splits) in both forms — that is base-level, filed as #1316, and not caused by #218.

What is left — exactly two things

1. The quant rebuild's conclusion, not its rows. The rebuild is measured for both models (quant/*/results-adapter.tsv): 7B Q4_K_M /
Q5_K_M / Q6_K / Q8_0 and 2.7B Q4_K_M-imx / Q6_K / Q8_0. But the 7B arm that matters moved off its bar:

7B, 32 chunks	merged form	adapter form
`Q5_K_M` size	5577 MiB	4938 MiB
KLD	0.019361	0.019383 (unchanged within noise)
top-1	95.025 %	94.792 % (−0.233 points)

pi-zquant's own recorded note on this model says "the margin by which Zamba2-7B-Instruct Q5_K_M clears the top-1 bar is 0.025 points" — the
adapter form moves top-1 by 0.233. So Q5_K_M no longer clears that bar, and the 7B sweet spot needs re-adjudication (their call and their bar):
Q6_K measures 96.593 % / KLD 0.010430 at 5829 MiB, which would clear it. That is a decision, not a measurement, and it is the reason this
issue should not close yet — closing now would bury it.

2. docs/model-families/zyphra.md (1bit-MONSTER repo, not the engine): last touched Sep 30, still carries none of these sizes
(5,328,903,360 / 14,722,123,360, the −2.005 GiB change, the --tensor-type ffn_gate_up_adapter=f16 note for a stale quantizer) and has no
open PR. Assigned to cl-1077 in my hand-off; flagged here so it does not get lost when the code issue closes.

Closure condition: a recorded sweet-spot conclusion in quant/Zamba2-7B-Instruct/ + CLAIMS.md (keep Q5_K_M, or move to Q6_K), and the
doc file updated from these numbers. The code needs nothing further — if you would rather track those two in their own items, say so and I will
move them and close this one on the table above.

Closing PR opened: engine #1345 — docs: a Zyphra family page — sizes, quantization picks, and the stale-quantizer trap (Refs #1036),
branch docs/zyphra-family-page @ 677af4e21. Docs-only, tools/copyright.py --check clean. I will close this issue when it merges, per
the orchestrator's instruction — everything else on the bar is already met (see the table in my previous comment), so nothing new is being
added here.

What the PR carries, mapped to the two items it was asked to cover:

The 7B pick moves to Q6_K (5829 MiB, KLD 0.010430, top-1 96.593 %), with Q5_K_M listed as the smaller option that misses the
bar (4938 MiB, 0.019383, 94.792 %, −0.21 points) — written into the page with the reason, because the reason is the interesting part:
the adapter form moved this model's top-1 down 0.52 points against 0.025 of headroom, while it moved the 2.7B's up 0.05. The
32-chunk rows are marked so nobody reads a 16-chunk number against a 32-chunk bar.
There was no engine-side registry/recipe pick to update, and I would rather say that than invent one: config/recipes.json holds
llama-server settings per model/route (no Zamba2 entry), models/pins.json pins published repos' files (no Zamba2-7B/2.7B entry), and
registry/{architectures,census,check_models.tsv,checked,significant,support_pins}.json carry no per-model quantization; a tree-wide grep
for pick/sweet_spot/default_quant/recommended finds nothing. The pick has only ever lived in PUBLISH-READY.md, which already
carries this exact decision (Q6_K sweet spot + smallest usable, ~~Q5_K_M~~ withdrawn), so I left that file alone rather than restate
another lane's record. The engine now records it in the family page, which was the missing piece.
The archived monorepo was not touched, and docs/model-families/ does not exist in the engine (docs/ is flat), so the page is
docs/zyphra-models.md — linked from docs/models.md so it is reachable from the models index.

Also on the page, because both have cost someone a run: the stale-quantizer trap (--tensor-type ffn_gate_up_adapter=f16, otherwise the
rank-128 pair is quantized, its B matmul leaves HRX on every hybrid layer and pp512 drops 463.66 vs 589.47 with no error), the layout
asserts that catch a transposed pair before a wasted PPL, and the -ngl 99 caveat that the tok-embd GET_ROWS is currently off HRX
(engine #1316, base-level, both forms) so no placement number from this family is quoted as "0 off" without that line.

docs: a Zyphra family page — sizes, quantization picks, and the stale…

Sweep (pi-verify) at `4480a6df` — both causes are in the pin and both models have their row; closing

My last comment on this issue said the 7B half had no row here. It has one, on fork #218 (the PR that implements cause 2), so
this closes on the owner's acceptance rather than on my reading:

Code, in origin/main's pin 67d8c42ec:

21c814079  convert/model: store Zamba2's per-layer MLP adapter instead of baking it (engine #1036 cause 2) (#218)
a7afabe3a  model/zamba2: find the stored shared block by attn_output, so one-block models with attention adapters load (#229)
6e42864a   (cause 1, already in the pin before both)

Acceptance rows.

row	result
7B BF16 shrinks	16,875,503,904 → 14,722,123,360 B (−2.005 GiB), 771 → 786 tensors; layout `ffn_up` ×2, `attn_q` ×2, adapters 13+13, `zamba2.ffn_gate_up_adapter_rank=128`; sha256 `58b26e9a13791fee…`
7B PPL / KLD vs the merged form	PPL 8.499977 vs 8.5028 (0.033 %), KLD 0.000095, same top-p 99.755 %, 16 × 512 wikitext — two builds give the same numbers
2.7B PPL / KLD	PPL 16.7937 → 16.795417 (0.0102 %), KLD 0.000067 ± 0.000002, top-p 99.510 %, calibration 0.000000; file 6,009,822,848 → 5,328,903,360 B
`test-llama-archs`	`zamba2 OK (0.00e+00)` + roundtrip OK; the synthetic zamba2 model declares the adapter form, so the test exercises the new tensors
HRX arms (2.7B, gfx1151)	`GGML_HRX_STRICT=1` exit 0 on all four arms, pp512/tg64 0.997/0.988
old files still load	published merged-form quants unchanged (2.7B Q4_K_M-imx PPL 19.4233, 7B Q5_K_M 13.9863 at 4×512)

The one clause with no evidence, stated rather than buried: the acceptance's last line — "Zamba2 quant arms are rebuilt once,
after this merges" — is a re-quantisation pass, not a code change, and I found no row for it. It has no issue of its own; if the
owner wants it tracked as work, it needs a row, otherwise it will be rediscovered as "the published Zamba2 quants still carry the
merged form".

Closing — both causes shipped, both rows measured, pick decided, docs landed

Merge proof. Cause 1 and cause 2 are in the engine's pin, the fork's follow-ups are merged, and the numbers below were each measured against
the file the change produces:

what	commit
cause 1 — shared attention/FFN block stored once per block	fork #203, merge `6e42864a6`
cause 2 — per-layer MLP adapter stored as a low-rank pair, `base + B(Ax)` in the graph	fork #218, merge `21c814079`
loader fix — sources found by `attn_output`, so one-block models with attention adapters load	fork #229, merge `a7afabe3a`
test fixture — `n_mamba_heads` set, so the convert test means the same thing on transformers 4 and 5	fork #232, merge `a4cbbdddc`
engine pin `third_party/llama.cpp` = `67d8c42ec`	contains `21c814079` and `a7afabe3a` (verified by `merge-base --is-ancestor`, not by reading the sha); `registry/architectures.json` records the pin. `a4cbbdddc` is not in it and does not need to be: #232 is a fork-side test fixture and the engine references it nowhere, so it rides in with the next bump
docs	engine #1345 merged `d5ea7987c` (`docs/zyphra-models.md`); #1356 corrects its `-ngl 99` caveat

The 7B row is at #issuecomment-6701038418:
16,875,503,904 → 14,722,123,360 B (−2.005 GiB), 771 → 786 tensors, ffn_up ×2, adapters 13+13, rank key 128; against the merged form's own
logits KLD 0.000095, top-p 99.755 %, PPL 8.5028 → 8.499977. The 2.7B row: 6,009,822,848 → 5,328,903,360 B, KLD 0.000067, top-p
99.510 %, and test-llama-archs zamba2 OK (0.00e+00) + roundtrip with the fixture on the adapter form.

The quant rebuild was measured for both models (quant/*/results-adapter.tsv): 7B Q4_K_M/Q5_K_M/Q6_K/Q8_0 and 2.7B Q4_K_M-imx/Q6_K/Q8_0.
Its conclusion moved the pick, which is the last decision this issue was holding: Zamba2-7B-Instruct Q5_K_M → Q6_K (5829 MiB, KLD
0.010430, top-1 96.593 %), with Q5_K_M withdrawn at 94.792 % (−0.21 points) — recorded in PUBLISH-READY.md and in the engine's
docs/zyphra-models.md, which now says why: the adapter form moved this model's top-1 down 0.52 points against 0.025 of headroom while moving
the 2.7B's up 0.05.

HRX placement, stated with its caveat: the adapter adds three nodes per hybrid layer (+27 at 2.7B = 9 × A + B + add) and all of them are
on HRX, GGML_HRX_STRICT=1 exits 0 on both forms at prefill and decode, with no throughput cost (pp512 0.997, tg64 0.988 on one binary). The
GET_ROWS that is off-device at -ngl 99 is #170's default for every model, not this family's residue — and #233 (in the pin) makes
LLAMA_INPUT_LAYER_ON_DEV=1 work for a tied embedding, so it is a switch rather than a floor if anyone needs it.

Nothing on this issue is left open. Closing.

docs(zyphra): cause 2 landed too - stop describing the per-layer adap…

Navigation Menu

Repository navigation

zamba2: shared attention block stored once per hybrid layer (2 distinct sets written 13x) — 7B BF16 is 22.0 GB where the checkpoint is 14.8 GB #1036

Measured

Cause 1 — the shared attention block, written once per hybrid layer (3.68 GB avoidable)

Cause 2 — per-layer FFN adapters baked into full FFN matrices (2.97 GB avoidable, bigger change)

Reproducer

Sweep (pi-verify) at `918f731a` — cause 1 independently confirmed, cause 2 is the whole residual

Sweep (pi-verify) at `0f45221e` — cause 2 is in main's pin; two rows are missing before this can close

The code bar is fully met and shipped

What is left — exactly two things

Sweep (pi-verify) at `4480a6df` — both causes are in the pin and both models have their row; closing

Closing — both causes shipped, both rows measured, pick decided, docs landed

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

zamba2: shared attention block stored once per hybrid layer (2 distinct sets written 13x) — 7B BF16 is 22.0 GB where the checkpoint is 14.8 GB #1036

Description

Measured

Cause 1 — the shared attention block, written once per hybrid layer (3.68 GB avoidable)

Cause 2 — per-layer FFN adapters baked into full FFN matrices (2.97 GB avoidable, bigger change)

Reproducer

Activity

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

Sweep (pi-verify) at 918f731a — cause 1 independently confirmed, cause 2 is the whole residual

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

Sweep (pi-verify) at 0f45221e — cause 2 is in main's pin; two rows are missing before this can close

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

The code bar is fully met and shipped

What is left — exactly two things

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

Sweep (pi-verify) at 4480a6df — both causes are in the pin and both models have their row; closing

bong-water-water-bong commented on Nov 7, 2026

Closing — both causes shipped, both rows measured, pick decided, docs landed

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Sweep (pi-verify) at `918f731a` — cause 1 independently confirmed, cause 2 is the whole residual

Sweep (pi-verify) at `0f45221e` — cause 2 is in main's pin; two rows are missing before this can close

Sweep (pi-verify) at `4480a6df` — both causes are in the pin and both models have their row; closing