Repository navigation
llama-context: the fit-probe warning passes a C string, so clang builds it (engine main is red without this) - #223
Closed
bong-water-water-bong wants to merge 541 commits into
Closed
bong-water-water-bong wants to merge 541 commits into
bong-water-water-bong wants to merge 541 commits into
Conversation
The generic TQ1_0 decoder computed g - 5 and p - 4 (and the decode lane l - 10) for every lane and selected the result away; with no-wrap index arithmetic that let the compiler drop the groups 0..4 branch. Select first, then subtract. TQ1_0 GET_ROWS and MUL_MAT n = 9 / 64 now pass (were ERR 2.4 / 130). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
… decode ggml_kquant_ref_value had no branch for TQ1_0 (34), TQ2_0 (35) or MXFP4 (39), so the kquant decode kernels had no reference to check them against. ggml_kquant_ref_value_1bit decodes one value of a 256-value group, written from ggml dequantize_row_tq1_0 / dequantize_row_tq2_0 / dequantize_row_mxfp4 and independent of motifs/dequant_1bit.loom (the E8M0 half scale is built from its f32 bits as GGML_E8M0_TO_FP32_HALF). TQ1_0 selects before it subtracts, so no-wrap index arithmetic cannot fold a region away. Six check cases against it: ggml_kquant_mul_mat_decode_f32 for TQ1_0 (with add), TQ2_0 and MXFP4, and ggml_kquant_mul_mat_decode_tokens_f32 for each (3 tokens). The MXFP4 cases fill bytes 124..129 so every scale is normal and between 2^-3 and 2^2. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: ADD_ID and SWIGLU_OAI on HRX (gpt-oss: 193 -> 49 graph splits)
… positive i32) 0x7C7C7C7C + 5 * 0x01010101 overflowed i32, so iree-test-loom refused the MXFP4 cases (OUT_OF_RANGE in the value materializer). Bytes 121..127 keep every E8M0 scale normal, between 2^-7 and 2^-1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: TQ1_0 and TQ2_0 weights (decode lanes, select-before-subtract fix, independent reference + check cases)
…= 512 tokens) prepare_flash_attention_value transposes V with ggml_copy_transpose_f16 once a prompt batch has >= 256 queries and >= 512 KV rows. That kernel reads `kv_heads x value_head_size`-wide contiguous rows, but MLA (deepseek2/GLM) stores V with the QK head-size stride, the layout the prefill matcher accepts since #95: 576 elements per row, of which V is the first 512. The transpose then read misaligned rows and attention used the wrong values. A V whose rows aren't contiguous now keeps the row-major layout; that kernel takes the real row stride (value->nb[1]). GLM-4.7-Flash Q4_K_M, llama-perplexity -c 512 -b 512, 4 chunks (transpose path): HRX0 before PPL 315,664 +/- 40,417 HRX0 after PPL 5.9160 +/- 0.479 CPU PPL 5.9592 +/- 0.484 Non-MLA models have contiguous V rows and keep the transpose: Qwen3.8-27B-Q4_0-H32 PPL 5.8714 before and after (4 chunks). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…foldable Q4_0 weights onebit.hadamard_q4_0 files (#58) are written by the engine's tools/hadamard_q4_0.py, which checks that every Q4_0 tensor it writes, token_embd aside, is rotated. The loader folded only the Q4_0 tensors on its whitelist and left the rest alone, so a rotated Q4_0 weight outside the whitelist ran without its activation transform. That would answer wrongly with no error. - ssm_ba (qwen3next's fused beta/alpha projection) joins the whitelist: qwen3next builds it through build_lora_mm (models/qwen3next.cpp), like ssm_alpha/ssm_beta in qwen35. - A stamped file with any other non-embedding Q4_0 weight off the whitelist is refused with the weight's name, not run. No existing file changes behaviour. The H32 files on hand, by Q4_0 weights the loader can't fold: Qwen3.8-27B (qwen35) 0 of 456, Qwen3-Coder-30B-A3B (qwen3moe) 0 of 337, Qwen3.6-35B-A3B (qwen35moe) 0 of 401. Qwen3.8-27B-H32 on HRX0: PPL 5.8714 before and after (wikitext-2, 4 x 512). GLM-4.7-Flash-H32 (deepseek2, 235 MLA weights not foldable) is still refused by the earlier arch check. No qwen3next H32 file exists to run. tools/hadamard_q4_0.py does not rotate ssm_ba yet, and its invariant check rejects any Q4_0 tensor it hasn't rotated. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: no V transpose for MLA strided V (GLM-4.7-Flash prompts >= 512 tokens: PPL 315,664 -> 5.92)
llama-hadamard: fold qwen3next ssm_ba; refuse stamped files with unfoldable Q4_0 weights
Assisted-by: Codex
ggml-hrx: let Loom select AMDGPU runtime globals
…andalone ggml-hrx: reuse Q5_K/IQ4_XS block metadata in SwiGLU decode
…uard) When gpt-oss's expert MUL_MAT_IDs run on the CPU while ADD_ID / SWIGLU_OAI run on HRX between those CPU splits, the result is wrong (KLD vs CPU 1.65, garbage text). The root cause is still open (engine ggml-org#286); this is a guard. moe-placement-guard.cpp (ours) claims ADD_ID / SWIGLU_OAI only when the MUL_MAT_ID they follow (directly or through ADD_ID) will run on HRX: HRX executes the op, and the expert weights sit in a buffer HRX can read. The second check matters for decode: when HRX declines the expert MUL_MAT_ID at load time, llama.cpp puts the experts in CPU_REPACK, and every MUL_MAT_ID then stays on the CPU, even the decode shapes that mul_mat_id.decode_f32_wave64 could run. One-line hook in device_supports_op. tests/test-hrx-moe-split.cpp: gpt-oss MoE block with MXFP4 experts in CPU_REPACK (and in a host buffer), biases and routing in HRX memory; checks the result against the CPU and the placement. Without the guard the CPU_REPACK pass gives nmse 0.058 at 1/3/12/40 tokens (the ggml-org#286 failure, reproduced in isolation); with it 1.5e-15. gpt-oss-20b MXFP4, KLD vs CPU (8 x 512): GGML_HRX_DISABLE_DISPATCH=mul_mat_id.f32_f32_wmma: 1.650 -> 0.031, greedy text correct default: 0.0294 before and after, same greedy text, pp512 / tg128 unchanged Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: keep ADD_ID / SWIGLU_OAI with their MUL_MAT_ID (placement guard, engine ggml-org#286)
…al requests gave different logits) Identical greedy requests in one llama-server process (cache_prompt false) gave different logprobs: Qwen3.8-27B UD-Q4_K_XL no MTP, --ctx-checkpoints 0, /completion top-5 x5: up to 0.042 between requests, top-5 ids changed at step 0. Qwen3-0.6B Q4_K_M in one llama_context with seq_rm between runs: 0.10-0.13 max |dlogit| at the first logits. llama_memory_clear(data = true) (KV zeroed) or GGML_HRX_DISABLE_DISPATCH=flash_attention made every run identical. Cause: both FA kernels multiply P = 0 for masked keys by V of those rows in an F16-accumulating WMMA, and the result depended on the sign of that V: flipping the sign of one masked V row (cell 11, first masked key) changed one output element by one F16 ulp. Rows past the sequence end hold whatever an earlier request (or a rejected draft) wrote, so the output depended on request history. F32 accumulation made this rarer (3/40 random cases instead of 12/40) but not zero, so the kernels now zero the V rows of masked keys: - decode split (1..15 query rows): per 16-key tile, every key visible -> direct V fragments (unchanged), no key visible -> zero V fragment, mixed -> V staged per subgroup in LDS with masked rows zeroed. The partial-last-block path zeroes masked rows in its existing staging. - prefill (16+ rows): blocks where some valid row masks some key stage V through LDS and zero keys that no row of the tile gives a probability; the KV tail path does the same. - tests/test-hrx-fa-masked-v: 27 cases (decode split and prefill, sequence end inside a block and in the last partial block); masked K/V rewritten five ways (random, +x, -x, +0, -0) must give bit-identical output, plus NMSE vs CPU <= 5e-4. Checks (strixhalo, balanced 85 W): - test-hrx-fa-masked-v: 27/27 pass x3 (before: 5/27 fail). - Random-input sweep, masked V +x vs -x: decode split 0/40 cases differ (before 12/40), prefill 0/40 (before 3/40). - Qwen3-0.6B, 5 runs in one context with seq_rm, and runs with a different 30-token request in between: bit-identical logits for 40 steps (before 0.10-0.58 max |dlogit|); same for a 280-token prompt through the prefill kernel (ubatch 64). - Qwen3.8-27B llama-server, no MTP, --ctx-checkpoints 0, /completion top-5 x5: 5/5 identical, equal to the first request of the old build (before: 5 different). - Qwen3.8-27B MTP n3 chat x8: 8/8 identical text (unchanged). - test-backend-ops -o MUL_MAT -b HRX0: 322/322 x2. iree-test-loom (hrx-system 244cd38) FA check cases with --sanitizer=access: prefill 6 cases and decode split case, masked_blocks, decode_513, tail pass (the old decode split kernel fails emission under the sanitizer for the 32-head cases; multirow_differential fails SUBRANGE/024 in its test-only helper before and after this change). Speed (llama-bench -fa 1, interleaved, 3 rounds, start <= 60 C, medians; balanced 85 W): - Qwen3-0.6B pp512 20815 -> 19835 (-4.7%), tg128 301.5 -> 301.0. - Qwen3.8-27B pp512 344.7 -> 345.3, tg128 10.16 -> 10.49 (both within run-to-run spread). - 0.6B decode device time (64 steps): 237.5 -> 241.6 ms; decode-split FA 9.34 -> 11.27 us. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: masked KV rows reach flash-attention P*V as V = +0 (identical requests gave different logits)
… FlashAttention A sink adds exp(sink - M) to the softmax denominator only, so with M and S the row maximum and sum of exp(scale q.k + mask - M): output_with_sink = output_without_sink * S / (S + exp(sink - M)). FlashAttention runs unchanged; ops/attention_sink_f32.loom + dispatch_registration/common/dispatch-attention-sink.cpp (ours) recompute M and S per (token, head) row with its conventions and rescale its output in place (a fully masked row is written as zeros). AMD kernels untouched; in dispatch-flash-attention.cpp the prefill matcher accepts the 5-input node and its dispatch appends the rescale, while the gate-fused and decode-split next_q8 variants keep the 4-input form (the latter publishes a Q8 copy of the output for the next projection). check.cases (prefill 3 x 300 keys, decode 1 x 1000, fully masked) vs a full-attention reference with and without the sink, 8 query heads over 2 KV heads, --sanitizer=access. gpt-oss-20b: graph splits 49 -> 1. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ut's storage The in-place guard compared value ids, so a different value (view) of the same allocation passed as disjoint (review). Compare storage (ValueMap::same_storage) and the byte ranges. tests/test-hrx-attention-sink.cpp: a 5-input FLASH_ATTN_EXT whose output is aliased onto the query's storage must be refused (fails with the old check); a disjoint one is accepted. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: attention sinks after FlashAttention (gpt-oss fully on HRX: 1 graph split)
…ode) ggml_dequant_weight_tile_bytes had no case for PQ2_0 (format 72) or PTQ1_0 (73) and returned 0. #62 added both to ggml_dequant_weight_row_bytes but not here. The low-token SwiGLU (ggml_mul_mat_swiglu_f32_f32_lowtoken_dot, 1-5 tokens) sizes a weight row as K/256 x tile bytes, so every row read row 0's weights and the output collapsed ("red color color color the the the"). PQ2_0 is two 34-byte blocks per 256 values (68), PTQ1_0 two 28-byte blocks (56). Only models whose decode SwiGLU reaches that kernel were hit. On Ternary-Bonsai-1.7B, the hidden-2048 qwen decode norm publishes a q8_1 copy of its output, kquant decode steps aside, and the gate/up pair lands in the low-token SwiGLU. The 27B files and TQ1_0/TQ2_0 took kquant decode. Decode-shaped llama-perplexity (-c 128 -b 128 -ub 1, wikitext-2): Ternary-Bonsai-1.7B PQ2_0, 4 chunks: HRX0 before PPL 1,225,624 +/- 189,778 HRX0 after PPL 20.6537 +/- 4.797 CPU PPL 20.6972 +/- 4.810 Ternary-Bonsai-2-27B PTQ1_0, 2 chunks, low-token SwiGLU forced (GGML_HRX_DISABLE_DISPATCH=kquant.swiglu): HRX0 before PPL 18,534.8; after 11.1754 (default routing 11.1756) Greedy, "The three primary colors of light are": before " red color color color color theit the the ..."; after " red, green, and blue. These colors are used to create all other colors through mixing." (= CPU). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
ggml-hrx: PrismML tile bytes for the low-token SwiGLU (PQ2_0 1.7B decode)
Add native Q1_0 x Q8_1 vector kernels with one-row and two-row schedules, and select Q8_1 publication for amortized token-one Q1_0 Q/K/V fanout. Keep generic F32 fallback behavior for aliased weights and non-beneficial consumers. Assisted-by: Chiron
Reuse each decoded Q1_0 weight tile across all 64 prefill tokens with a four-wave schedule while retaining the legacy route as an A/B control. Preserve source-role-aware model benchmark linking so the model-derived route remains reproducible. Validated on Bonsai pp512 with 3.13% lower latency and 3.14% higher throughput. Assisted-by: Chiron
Broaden the shared workload declaration so skinny and tiled providers can coexist behind compile-time provider selection while retaining the skinny route's existing constraints. Assisted-by: Kronk
Add shared direct-lookup vector and tiled routes for IQ1_S, IQ1_M, IQ2_XXS, IQ2_XS, and IQ3_XXS. Reuse immutable codebook resources by content while preserving dtype-specific decode semantics and generic fallbacks. Assisted-by: Codex
Reuse the published Q8_1 activation while decoding Q1_0 gate and up projections together. This reduces duplicate activation work in token-one Bonsai decode while retaining the generic fallback and route guards. Assisted-by: Chiron
Validated across Qwen 27B, Qwen 30B, and the Lemonade model cohort. Assisted-by: Codex
Extract IQ4_NL nibbles directly from the packed word and schedule the block-scale load earlier. This improves the shared paired, tiled, and vector paths while preserving existing route and ABI contracts. Assisted-by: Codex
test: update HRX RMSNorm JIT corpus coverage
The f32_f32 WMMA cores carried their whole K accumulation in an f16 accumulator vector, which lowers to v_wmma_f16_16x16x16_f16. On gfx11 that instruction's accumulator keeps about 8 mantissa bits, and a long sum of small products cannot even be represented: every 16-wide step of 0.0234375 products adds 0.5 instead of 0.375 once the running sum passes 32. Reproduced from engine ggml-org#284 with MXFP4 weights of 0x77 (0.0234375) and 1.0 inputs, output_size 96, 1 token, 1 route: K exact f16 accumulator 2048 48.0 53.3125 2304 54.0 61.3125 2816 66.0 77.3125 (+17%) 2880 67.5 79.3125 A bf16 accumulation of the same 16-wide steps reproduces those numbers exactly, so the f16 accumulator is the root cause and the fix is to use the f32 accumulator of the same 16x16x16 WMMA. gfx11 wave64 f16->f32 WMMA uses a 4xf32 accumulator payload (rdna3_wmmar3_f32_16x16x16_f16_w64, v_wmma_f32_16x16x16_f16), so each core's accumulator becomes vector<4xf32> and the finished tile is staged in f32. The publish path already consumed f32, so the f16->f32 widen is gone. Covers the cores engine ggml-org#284 names plus the ones that share the motif: motifs/mul_mat_id_f32_f32_wmma_core.loom common.mul_mat_id.f32_f32_wmma motifs/mul_mat_f32_f32_wmma_core.loom common.mul_mat.f32_f32_wmma ops/mul_mat_swiglu_f32_f32_wmma.loom common.mul_mat_swiglu.f32_f32_wmma ops/mul_mat_id_swiglu_f32_f32_wmma.loom Verified by linking and compiling every affected export for gfx11-generic and disassembling the artifact: each v_wmma_f16_16x16x16_f16 became v_wmma_f32_16x16x16_f16 (x8 in the matmul cores, x16 in the swiglu cores), and every export that compiled before this change still compiles.
…256 (engine ggml-org#991) (#199) The dense MUL_MAT admission required input_size % 256 == 0, and F16/BF16/F32 weights fall into the block-quantised arm of that predicate, so a model whose hidden size is not a multiple of 256 lost every dense matmul to the CPU while the rest of it stayed on the device (Zamba-7B-v1 n_embd 3712: 383 of 5113 nodes; BlackMamba-2.8B 1472: 145 of 1179). The refusal was policy, not the kernel's contract: the wmma kernel declares multiple_of(input_size, 32) and its core already rounds the contraction up to whole 256-blocks and masks both operands past input_size, so a K tail contributes zeros. Only element-wise formats can be unaligned at all - block formats are laid out in 256-value blocks, and a quantised tensor with ne[0] % 256 != 0 cannot exist in ggml - so the relaxation is one predicate about the element-wise formats. The relaxation is an explicit flag on the shared matcher, passed only by the three callers that dispatch the plain %32 kernel (plain, fused-unary and decode). The fused motifs that genuinely need 256 - mul_mat_postops, mul_mat_id_f16_f16, llm.attention_qkv, q6_K packed/shortlist, symmetric i8 - keep the old predicate: had the shared one been relaxed, a fused match would have compiled K=3712 against a kernel whose declared constraint forbids it. With the flag a fused match simply does not engage and falls through to the plain matmul, so the node stays on the device and only the fusion is lost. gfx1151: Zamba-7B-v1 383 -> 254 of 5113 nodes off (MUL_MAT 281 -> 165, UNARY 13 -> 0, CONT 89 unchanged), splits 384 -> 179; Zamba2-2.7B (2560) unchanged at 0 off; test-backend-ops -o MUL_MAT 404/404 and -o MUL_MAT_ID 852/852 on HRX0 vs CPU; greedy output byte-identical to the pre-change run. Not closed: the remaining nodes are the K % 32 != 0 class (ne=[264,512,2,1] etc.) and the %256 fused/MoE-id motifs. Co-authored-by: bong-water-water-bong <bong-water-water-bong@users.noreply.github.com>
`-fcd/--fused-cross-device` (#194) is declared with `set_examples({..., LLAMA_EXAMPLE_BENCH, ...})` in common/arg.cpp, but llama-bench parses its own closed argument set and rejects anything not in its whitelist, so the flag is unreachable from the one tool that measures the placement it changes: $ llama-bench -m model.gguf -fcd -p 512 -n 32 error: invalid parameter for argument: -fcd That leaves the tool unable to reproduce the configuration #194 exists for (partial offload, where the fused op's layer stays on the CPU). Carry the flag the way the neighbouring placement flags are carried: a `cmd_params` field with the same default the library uses (off), the pair of switches, the instance field, `cparams.fused_cross_device` in `to_llama_cparams()`, and a results column that only appears when it is not the default. Verified on gfx1151 (HRX0), Qwen3-0.6B-Q4_K_M, -ngl 99 -p 64 -n 8: before: "error: invalid parameter for argument: -fcd" after : pp64 7497.98 t/s, tg8 361.93 t/s, "fused_cross_device | 1" in the header Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Stacked on the Zamba2 PR: Zamba v1 needs LLM_TENSOR_SSM_MIX / llama_layer::ssm_mix, which that PR adds. Forward-ported from the removed Vulkan line (1bit/vulkan-upstream @ 5319c0e) onto this base. The arch: 76 layers, `layers_block_type` alternating `mamba` and `hybrid`. A Mamba-1 layer is RMSNorm -> Mamba-1 -> residual. A hybrid layer is concat(hidden, input embeddings) -> RMSNorm (ATTN_POST_NORM, 2*n_embd wide) -> one shared transformer block (MHA with no rotary, k/v length = 2*n_embd/n_head, GEGLU FFN) -> per-layer linear mix (SSM_MIX) -> + residual -> RMSNorm -> Mamba-1 -> + residual and the transformers Mamba-1 mixer is not the plain one: `in_proj` is split into `n_mamba_heads` (2 here) heads, each with its own `x_proj`/`dt_proj` ({n_heads, dt_rank+2*d_state, head_dim} and {n_heads, head_dim, dt_rank}), so B and C enter the scan as groups and the matmuls run batched over the head axis. `build_zamba_mamba()` is `build_mamba_layer()` with those two matmuls replaced. The shared block is stored once in the checkpoint and every hybrid layer points at the same tensors (`shared_transf` under the first hybrid layer), so the model, the converter and the saver all have to handle the aliasing: `llama_model_saver` now remembers the tensor objects it wrote and skips a second visit (a *distinct* tensor with a colliding name still asserts), which is what makes save/load roundtrip pass. Tensors: A = -exp(A_log) reshaped to {d_state, d_inner}; dt_proj_bias and D flattened from {n_heads, head_dim}; in_proj's [x, gate] rows de-interleaved (the reference does view(d_inner, 2)); `ssm_x`/`ssm_dt` keep their head axis. The tokenizer is LlamaTokenizer from tokenizer.json, whose name list carries a [PAD] the embeddings have no row for, so the vocab is truncated to the model's 32000. `ssm_n_group` is written as the Mamba-1 head count, and the saver restores it from `ssm_x->ne[2]` because hparams keeps it at 0. Proof (CPU, GGML_HRX=OFF), Zyphra/Zamba-7B-v1 @ b8c9e7ede2f6: - test-llama-archs: `zamba ... OK (0.00e+00) ... roundtrip OK` (needs the 2*n_embd key/value length for the synthetic model, exactly as Zamba2 does). - converted by this branch: 784 tensors, 14.5 GB bf16. - llama-perplexity, calibration window, -c 512 --chunks 1: PPL 6.1107. - KLD against HF transformers `ZambaForCausalLM` (f32 both sides, the identical 255 positions of the same 512-token window): KLD(HF||GGUF) mean 0.0022 (p99 0.0036, max 0.0036), same top-1 100.000 %, total-variation distance 0.0065. Co-authored-by: pi-zarch <pi-zarch@1bit.gg> Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
…t use it where it can (engine ggml-org#900) (#200) The residency budget was never compared with the set of host weights a run needs: placement tested each weight against its file size, acquire() charges the materialized length (a carrier the dispatch builder picks after placement: 132 B per row-block for the shared4 layout against 136 B in the file for IQ4_XS, 144 B for the row64 ones), and the arena's advice is a need+resident+parked snapshot that moves with the budget - following it cycles 38 <-> 44 MiB on a 4B. Now the sum is taken before the first request: one entry per distinct (identity, generation), keeping the largest materialized length seen for each, so the total is the model's set and not one program's slice of it. When it exceeds the budget, one line says so; placement declines host-weight ops when the set alone cannot fit. The other direction is deliberately not inferred: a set that fits does not mean a run that fits, because the arena also holds the slots parked while programs are in flight. Measured on Qwen3-4B-IQ4_XS at -dev HRX0 -ngl 0 -fit off: the set is 304.3 MiB (579 program observations, GGML_HRX_HOST_WEIGHT_TRACE=1), while the host-weight path only runs at a budget of 3072 MiB or more - 2048 MiB fails with 505.1 MiB resident and 1448.6 MiB parked. The number is a lower bound, and the message says so; the parked-slot accounting is engine ggml-org#900's remaining subject. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
ggml-org#900) (#202) resident + parked is bounded by the arena and the arena by the limit, so the snapshot of a capacity miss is always limit + one weight. Printed as advice that is the measured 6 MiB-at-a-time climb (2304 -> 2310 on a run whose floor is 2560) and, below the budget in force, the measured 38 <-> 44 MiB cycle. The capacity branch now names the numbers and says the snapshot cannot size the run; the fragmented wording (#180) is untouched and the unbounded arm no longer prints limit 0.0 MiB. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
TZHA-BUILD stage 1 needs the exact q/k the model's full-attention layers see, from our own
quantised weights, to measure how concentrated dense attention really is before any ZHA
kernel is written. There was no way to get them: ggml has no generic activation dump and the
graph's cb() hook is only reachable from inside the library.
Adds src/llama-zha-dump.{h,cpp}: off unless LLAMA_ZHA_DUMP_DIR is set, in which case the
tensors llm_graph_context::cb() reports as Qcur/Kcur (post-RoPE) and Qcur_normed/Kcur_normed
(pre-RoPE) are marked as graph outputs and appended, after the ubatch computes, to one f16
file per (layer, name) with ubatch/token manifests for alignment. Strided views are read
row-wise with nb[1]/nb[2]: qwen35moe's Qcur is every other head_dim block of a [q|gate]
projection, so a flat read returns the interleaved rows. A ubatch whose tensor set differs
from the first complete one is skipped whole, so no file can drift out of alignment.
No behaviour change when the variable is unset: one bool check per cb() call, which is what
made it possible to keep this out of the hot path.
Measured (Qwen3.6-35B-A3B UD-IQ4_XS, CPU build, 75,251-token prompt, 10 full-attention
layers): 13 GB of dumps, all 40 tensors row-aligned; the dumped pre-RoPE q/k reproduce our
own RoPE convention to 2.0e-4 relative (f16), which is what pins the half-split pairing,
n_rot=64, theta=1e7 this model actually uses.
Co-authored-by: pi-zha <zha@1bit.local>
…visible (engine ggml-org#674) (#174) * hrx: log a residency decline, so a budget that reshapes the graph is visible (engine ggml-org#674) host_weight_exceeds_residency() decides placement in device_supports_op, and it was silent: a budget that moved nodes off HRX looked identical in the log to one that changed nothing, because the only witness was the GGML_HRX_REPORT placement census. A placement decision that can change the graph has to be greppable. One line per distinct (weight, op) pair, eight pairs, then one line saying the rest are suppressed - de-duplicated because device_supports_op is queried repeatedly per node (reserve and compute passes), so a plain cap spends the whole budget on the same decision (measured: 8 lines for 2 nodes at a 128 MiB budget, and at 8 MiB the 109 declined tensors would never have been visible). The predicate stays pure and behaviour is unchanged - measured on the pin, Qwen3-4B -ngl 0, ggml-org#715 trio, one gpu-hold ticket per arm set: budget unset -> 0 lines, census 288 off HRX, `Okay, the user is asking`, +4055M GTT budget 128 -> 2 lines (GET_ROWS + MUL_MAT on token_embd, 304 MiB > 128 MiB), census 290, `Compute error.` budget 8 -> 8 lines + suppression notice, census 469 off HRX (MUL_MAT 181, ADD 35, GLU 36), `Compute error.` The header records the invariant this makes testable: no placement change may reintroduce the weightless-op/weight-op split. At this pin the bisected corrupting pair {MUL_MAT, MUL} is not expressible at any whole-MiB budget (norm scales are 512 B - 10 KiB, the knob's floor is 1 MiB) and every budget that declines a node also refuses to run, so no declined placement executes. Instrument only - no fix and no behaviour change; engine ggml-org#674's row is the owner's to close. * hrx: say where the placement guard can fire, and why it can be silent (engine ggml-org#674, PR #174 review) Comment only. The review's finding is right for the command the PR printed: on llama-cli at its default verbosity the host-weight arena refuses the model during load (1.2 s in), device_supports_op is never asked, and the guard that this PR adds says nothing. The same binary at -v or -lv 2 reserves at 0.17 s, logs the two 128 MiB lines, and reaches #173's refusal later at the first execution (8.99 s); 3/3 runs each way, both ending in Compute error. So silence in that log is not evidence that no node moved - it can mean the run never got as far as the split, and the reviewer's arm and mine were the same build measured at two points of one race. Also notes the process-static dedupe (a second model in the same process logs nothing), per the same review. --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
… (engine ggml-org#1036) (#203) Every hybrid layer of a Zamba2 model runs the same transformer block, and the checkpoint stores it only num_mem_blocks times (twice at 7B), but the converter wrote one copy per hybrid layer -- 13 copies of two blocks, which is why our 7B BF16 is 10.977 B params / 20.45 GiB for a 7.357 B checkpoint. Write each block once, at its source layer, and alias the other layers to it in the loader. - conversion/zamba2.py: a block tail that this checkpoint stores no per-layer adapter for is written once per block instead of once per layer. The shareable set is derived from the checkpoint's own adapter tensors, so a tail with an adapter (ffn_up, whose MLP adapter is baked per layer) stays per layer, and a checkpoint that adds attention adapters needs no code change. - src/models/zamba2.cpp: the shared tensors are TENSOR_NOT_REQUIRED, and after the layer loop a hybrid layer that carries no copy aliases attn_post_norm/wq/wk/wv/wo/ffn_norm/ffn_down from sources[visit % num_blocks]. The sources must be the first hybrid layers of the file, which is asserted. A file with a copy in every layer -- every file written before this change -- makes every layer a source, so each alias is the identity and the old path is unchanged. 7B-Instruct: 848 -> 771 tensors, 10.977 B -> 8.433 B params (20.45 -> 15.71 GiB BF16). The remaining 11 gate_up copies need the adapter applied in the graph, which is cause 2 of ggml-org#1036 and stays open. Co-authored-by: agent <agent@local>
…AT_ID (engine ggml-org#991) (#204) BlackMamba-2.8B loses 90 of 1179 nodes to the CPU and 72 of them are one MoE block pattern: MUL_MAT_ID x 54, GLU x 18. Its experts are [1472, 3872, 8] and [3872, 1472, 8], so the contractions are 1472 and 3872 -- both multiples of 32, neither a multiple of 256. common_mul_mat_id_supported_dense_input_size() forwarded to the dense table, whose default branch is % 256, and F16/BF16/F32 are not among its listed cases -- so element-wise expert weights inherited the block-quant default and were refused. The refusal is policy, not hardware: the three matchers that consult this predicate bind five kernels and all five declare or assume multiple_of(32), and the wmma core already pads the contraction (padded_input_size = input_size + 255) with a valid_k gate and masked activation loads. Route the predicate through common_mul_mat_supported_plain_dense_input_size(), the same element-wise/block split the plain dense path took in #199, so only F16/BF16/F32 verdicts change. Adds three test-backend-ops cases at the real shapes next to the existing gpt-oss k=2880 ones. Co-authored-by: agent <agent@local>
…-org#1078) (#205) With a device that claims host buffer types, `-ngl 0` silently loses Flash Attention. ggml-hrx's device_supports_buffer_type() returns true for every host buffer (ggml_backend_buft_is_host), so ggml_backend_sched's pass 3 -- "set remaining unassigned nodes to the backend with the most supported inputs", lowest id winning a tie -- hands it the FLASH_ATTN_EXT node, whose inputs at -ngl 0 are all CPU activations. The node is unassigned in pass 1 because the scheduler skips FLASH_ATTN_EXT there (the sinks tensor is too small to choose a backend by). resolve_fused_ops then compares that backend with model.dev_layer(il) = CPU, and on the mismatch sets cparams.flash_attn = false for the whole context: W resolve_fused_ops: layer 0 is assigned to device CPU but Flash Attention is assigned to device HRX0 W resolve_fused_ops: Flash Attention not supported, set to disabled Name the fused attention tensor and pin it to the layer's own backend in graph_get_cb, the same shape as the existing CPU pin for the non-FA path. For -ngl 0 that device is the CPU, so FA stays enabled and runs where the K/V cache already is; for an offloaded layer the pin names the device the scheduler was already choosing. The pin matters rather than "just leave FA enabled": FA forced onto HRX0 over CPU-resident weights returns a different token outright (-3.089767, `air`, and `airc.' 33.. 333..` as an essay), so llama.cpp's refusal is right -- it is the placement that is wrong. Measured, Qwen3-4B-IQ4_XS, -ngl 0, -c 512 -b 32 -ub 32, n_predict 1, n_probs 8, greedy, one gpu-hold, before this commit -> after: -dev none : FA enabled, -1.084810 -> unchanged, -1.084810 (no CPU-route change) -dev HRX0 : FA disabled, -1.370574 placement 1446 nodes, 288 off HRX (RMS_NORM 144, MUL_MAT 72, CONT 36, SOFT_MAX 36) -> FA enabled, -1.369900 placement 1266 nodes, 180 off HRX (RMS_NORM 144, FLASH_ATTN_EXT 36) 0.6B q4_K_M -dev HRX0: -1.550458 -> -1.552646, FA enabled 0.6B f16 -dev HRX0: -1.818972 -> -1.817940, FA enabled 0.6B q4_K_M -dev HRX0 -ngl 8: FA enabled, 986 nodes, 105 off HRX (RMS_NORM 84, FLASH_ATTN_EXT 21), first token -1.562199 -- the offloaded route keeps its offloaded FA nodes. This does not make -dev HRX0 equal to -dev none (0.285 nats remain): that is the other half of ggml-org#1078, the 60-80% of nodes the scheduler puts on HRX0 over host buffers, which is a placement-policy decision and not touched here. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
… published GGUFs have an authoring site (engine#1098) (#206) * convert: Zamba2-VL (Zamba2 language model + Qwen2.5-VL tower), so the published GGUFs have an authoring site The three published `1bit-MONSTER/Zamba2-VL-*` GGUFs carry the template `9df322391abffe14587e6f3cfaea42420eaffec364835debfcb13c0e4c6eeab6` (1045 B), and the code that wrote it exists only in the Vulkan-era checkout (`llama.cpp-vulkan/conversion/zamba2vl.py`, now removed with that build, RFC #213) — so a pin bump could not re-derive those files (engine ggml-org#1098). This brings the module to the pinned tree, unchanged and registered: - `conversion/zamba2vl.py`: `Zamba2VLTextModel` (the `language_model.` half, `MODEL_ARCH.ZAMBA2`) and `Zamba2VLVisionModel` (the `vision_tower.` half as a Qwen2.5-VL `MmprojModel`), i.e. the model + mmproj pair the repos ship. `_CHAT_TEMPLATE` is Zyphra's own multimodal format — `<s>`-delimited turns, `<|vision_start|><image><|vision_end|>` — with the media markers as llama.cpp's `<__media_<id>__>` bridge, which is what makes the file usable by mtmd. - `conversion/__init__.py`: `"Zamba2_VLForConditionalGeneration": "zamba2vl"` in both `TEXT_MODEL_MAP` and `MMPROJ_MODEL_MAP`, so `load_all_models()` imports it like every other family. Verified against the pinned tree (no weights needed): both classes register for `Zamba2_VLForConditionalGeneration` (`ModelType.TEXT` -> `Zamba2VLTextModel`, arch `MODEL_ARCH.ZAMBA2` = 67; `ModelType.MMPROJ` -> `Zamba2VLVisionModel`, arch 1), and the module's `_CHAT_TEMPLATE` hashes to `9df322391abffe14…`, 1045 B — byte-identical to the template in all three published files. Pre-existing, untouched: `load_all_models()` at this pin already warns for `cambrian`, `deepseek41`, `flint`, `qwen4exp` and `dots3` (`Missing property 'model_arch'`, `no attribute 'example'`, a missing module); this change adds no warning. Engine side picks this up with the next llama.cpp pin bump; nothing in the engine changes here. * convert: keep Zamba2-VL's top-level architecture for the text pass, or the VL class is never built Registering `Zamba2VLTextModel` is not enough: `get_model_architecture()` prefers `text_config.architectures[0]` for the text pass, and Zamba2-VL's `text_config` says `Zamba2ForCausalLM`. A conversion therefore built the **plain** `Zamba2Model`, so the ported `set_vocab` never ran and the output carried Zyphra's `chat_template.jinja` (74475f26d1…, 796 B) and an unpadded 32004-token vocab instead of the published 1045-byte template and 32064 tokens — a silent wrong artifact, which is exactly what engine#1098 is about. `conversion/base.py`'s TEXT exception list already exists for this shape of model (`StepVLForConditionalGeneration`, `Sarashina2VisionForCausalLM`, `Exaone4_5_…`, `Step3p7_…`); it now names `Zamba2_VLForConditionalGeneration` too, with the reason in a comment. Measured on `Zyphra/Zamba2-VL-1.2B`, converted end to end through this branch and compared with the published `1bit-MONSTER/Zamba2-VL-1.2B-GGUF/Zamba2-VL-1.2B-Q8_0.gguf`: | | published | before this commit | after | |---|---|---|---| | `tokenizer.chat_template` | `9df322391a…` (1045 B) | `74475f26d1…` (796 B) | **`9df322391a…` (1045 B)** | | tokens | 32064 | 32004 | **32064** | | architecture | zamba2 | zamba2 | zamba2 | | tensors | 398 | 378 | 378 | The 20-tensor difference is the pin's own #203 ("store Zamba2's shared transformer block once per block"): the published files predate it and carry `blk.11.*` / `blk.17.*` a second time. It is not a port difference, and a re-quantise from this tree would not reproduce those 20 tensors. --------- Co-authored-by: agent <agent@local>
…ml-org#991) (#207) A MUL_MAT whose activation carries batch planes against an unbatched weight is the same 2-D contraction repeated per plane, and it is how the SSM/Mamba ports write their small projections. Every format-agnostic matcher refused it in common_match_mul_mat_any_format on common_is_2d(*input), before any size rule, so those nodes fell back to the CPU and split the graph. This is not a kernel gap: the two kernels involved (ggml_mul_mat_f32_f32_wmma for prefill, ..._decode_wave64 for token 1) already compute a 2-D tile from a buffer base, so the planes are bound one dispatch at a time. dispatch-gated-delta-net.cpp already binds { id, t * nb[2], byte_count - t * nb[2] } per token, which is the same mechanism. - common_match_mul_mat_any_format takes allow_batched: the activation and output may carry planes when the output's planes match the activation's. ggml pairs the operands' planes when both carry them and broadcasts when only the activation does, so the weight may be one matrix shared by every plane or one matrix per plane (with the same (dim2, dim3) factorisation, since pairing follows that index order). plane_count and batched_weight ride on the match. - common_plane_offset maps a plane index to its byte offset in a contiguous value. - build_batched_mul_mat_dispatch emits one dispatch per plane: activation, weight and output bindings offset to the plane, the token count is the per-plane one. - Two registrations, common.mul_mat.f32_f32_wmma_batched (82) and ..._decode_batched (62), one above each plain dense registration they mirror. The K contract is unchanged: still the plain element-wise predicate, so K % 32 != 0 stays on the CPU until the padded tile lands. Test cases are the shapes the port actually emits (m=264,n=8,k=3712 and m=464,n=8,k=256 with a shared weight; m=264,n=8/1,k=3712 with a BF16 matrix per plane; plus the unaligned k=232 rows that must stay refused). Co-authored-by: agent <agent@local>
…(engine ggml-org#991) (#208) The element-wise weight readers take four values at a time and the activation load is masked with `vector.mask.range [input_k to input_size)`, so a contraction only has to be a multiple of 4 for every packet to stay inside its weight row. The 32-value packet (and the 256 floor) the kernels declared was a contract nobody's arithmetic needed, and it is what leaves the Zyphra ports' dt_proj on the CPU: BlackMamba ssm_dt.weight is [92, 2944] and Zamba v1's is [232, 3712]. - common_mul_mat_supported_plain_dense_input_size: element-wise formats now need input_size >= 4 && % 4 == 0. Block formats are untouched and keep their %256/%128/%64/%32 from common_mul_mat_supported_dense_input_size(). - mul_mat_f32_f32_wmma's ggml.mul_mat.input_size config and the assume in mul_mat_f32_f32_wmma_core and both entry points of mul_mat_f32_f32_decode: range 256 -> 4, multiple_of 32 -> 4. Weakening the contract is safe for every caller: the dispatcher still gates each format at its own granularity, and a block-format contraction is always %256, hence also %4. K % 4 != 0 stays refused on purpose: the last packet of such a row would read up to three values past the row, which is correct by accident only where the following row exists. Co-authored-by: agent <agent@local>
…d back (engine ggml-org#1078) (#209) The zha q/k dump (src/llama-zha-dump.cpp, #179) marks the tensors it wants with ggml_set_output() and reads them back after graph_compute. On the HRX route its pre-RoPE half came back as zeros - every layer, every row - while the post-RoPE half was fine: -dev none : L0 qpre max 10.41, rms 1.46 (values) -dev HRX0 -ngl 99 : L0 qpre every value 0.0 (zero-filled buffer) -ngl 99 DISABLE=fused: L0 qpre rms 14.66 (kpre), values back (the fusion is the difference) Cause: ggml-hrx treats a GGML_TENSOR_FLAG_OUTPUT value as external, so a fused match writes its *root*'s output for the caller (graph.cpp:tensor_is_external), but nothing does the same for the nodes a match *covered*: their only copy stays inside the fused program and their buffer is never written. So a match is now declined when it would cover a node whose tensor carries the read-back marker, and the plan falls through to a narrower registration that writes the node itself (one GGML_HRX_LOG_DISPATCH line per decline, 8 then a suppression notice). The marker is the explicit name prefix llama_zha_dump applies ("zha-readback.<name>"), not the bare GGML_TENSOR_FLAG_OUTPUT: llama.cpp marks several tensors as outputs for API reasons (result_norm is t_embd for the embeddings API), and honouring the flag alone declined the final-norm fusion in every ordinary run - measurable in the plan, and not something to trade for a debug tool. With the prefix, nothing outside a dump run carries it. Measured on the pin (Qwen3-4B-IQ4_XS, -dev HRX0 -ngl 99, one gpu-hold ticket per arm set, same prompt): ordinary run : 0 declines, first-token logprob -1.3745098, top-5 identical to 6 dp to the build without this change - i.e. the plan is the one it had before dump run : 8 declines naming zha-readback.Qcur_normed-N / Kcur_normed-N, qpre L0 rms 1.5323 and kpre 14.6636 - the GGML_HRX_DISABLE_DISPATCH=fused reference reads 1.5322 / 14.6648, so these are the real values and not a different approximation - and no "unsupported HRX node" A dumped intermediate with no fallback registration now fails loudly instead of writing zeros; that is the trade a debug tool should make. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…the CPU (engine ggml-org#991) (#210) * tests: pin the padded shapes at the token count the prefill ubatch uses (engine ggml-org#991) The first version of the padded-tile tests used n=8, and that was not enough: BlackMamba's prefill ubatch runs these contractions at 512 tokens. The predicate itself is fine there (both cases are OK on the device against the CPU reference), so the residue that GGML_HRX_STRICT=1 reports at 512 tokens is a different gate -- the model's dt_proj activation is a non-contiguous view at that ubatch size, which the matcher refuses before any size rule. Keeping the coverage anyway: it pins the token-count half of the contract, and it is what turned an assumption into a measurement. * hrx: densify a row-padded activation once instead of falling back to the CPU (engine ggml-org#991) The SSM ports hand their small projections a column slice of a wider tensor: BlackMamba's `dt_proj` contracts 92 of the 264 columns of `x_proj`, so its activation is a view whose rows are dense but gapped. The plain dense matcher refused it on `!input->contiguous`, which put all 18 `dt_proj` nodes of the prefill ubatch on the CPU (GGML_HRX_STRICT=1 aborts on `node_31`, 18 of 1179 nodes off, 37 graph splits). The kernels bind a packed 2-D tile, so the view cannot be bound directly -- but it can be copied once. The copy path already exists for `GGML_OP_CPY` (`ggml_copy_f32` / `ggml_copy_strided_source_f32`, which takes the source extents and strides as config), so this exports that as `append_f32_densify_dispatch` and uses it from the dense matmul builder: - `common_match_mul_mat_any_format` takes `allow_row_padded_activation`; the input may then be a row-padded F32 view (`nb[0] == sizeof(float)`), and the match records `strided_activation`. - `build_mul_mat_dispatch` allocates a packed transient for exactly that node, emits the densify dispatch, and binds the transient as the activation. One copy of K*tokens floats per node per ubatch (188 KiB here), not a per-layer fallback. - The plain and unary dense matchers opt in; everything else keeps requiring a contiguous activation. Test cases build the model's shape directly (`test_mul_mat_row_padded_activation`: dense weight, activation view of a 264-wide backing at K=92 and K=232, at 512 and 8 tokens). --------- Co-authored-by: agent <agent@local>
… (engine ggml-org#572) (#211) GGML_HRX_HOST_GAP=1 decomposed graph.execute down to the replay.* phases and nothing else. On the product path (Qwen3.8-Flash-Next through llama-server) 90-94% of wall sat inside graph.execute but outside every phase, and two identical-argv windows that differ 1.87x in prefill carry 409 of their 419.6 s difference in exactly that unaccounted region. The counters could not say which part moved, so the biggest phase on the slowest path was one number. Adds prep.* phases to the prepared path and to per-shape preparation: prep.prepare prepare_command_program at all four call sites: the two prepared-cache paths (prepared-command-program-cache.cpp:105, :175), the graph-program-cache miss (graph-program-cache.cpp:490) and the direct/uncached entry (command-program-executor.cpp:2262, execute_command_program) prep.rebind rebind of host staging + host weights prep.transient, prep.const, prep.upload, prep.init, prep.main, prep.download, prep.sync Each is one HostGapScope; off, they compile to a cached `if (enabled)` like the existing phases. Measured, Qwen3-4B-Q4_K_M on HRX0, llama-bench -p 512 -n 0 -r 3, GGML_HRX_HOST_GAP_VERBOSE=1: before graph.execute 7798.5 ms of 7823.6 ms wall, replay.total 1430.8 ms, unaccounted 6367 ms (81.4% of wall), no prep.* line after graph.execute 1855.7 ms, replay.total 671.6 ms, prep.prepare 1184.1 ms (1x), unaccounted 0.0 ms; per graph the three prepared executions are 169.2/171.0/169.6 ms with execute == replay.total == replay.wait The accounting closes, and the prepare cost is one-off per graph shape (1 of 4 graphs), not paid per execution. The two runs also differ in prepare time (6367 ms vs 1184 ms); that difference is not attributed here -- a warm kernel/graph cache is the candidate and the phase that now names it is the point of the change. What llama-bench can and cannot show, because it cost a review round to find out: on that harness only prep.prepare fires. The other seven phases are in bind_and_execute_prepared_command_program (command-program-executor.cpp:1849), while llama-bench goes through bind_and_launch_recorded_command_graph (:1949) and therefore reports the pre-existing replay.* phases instead. A benchmark run that shows prep.prepare and no prep.main is correct, not broken; the prepared-path phases need a llama-server window to exercise. Refs engine ggml-org#572 (instrumentation; it does not close the prefill gap). Co-authored-by: pi-fnprefill <pi-fnprefill@1bit.local> Co-authored-by: agent <agent@local>
…lling the decoder (#212) `ONEBIT_KV_SHARE=1` makes a context allocate its KV as exportable dma-buf chunks. Two problems turned a decoder that simply cannot export into a decoder that does not start, which is fatal for a hybrid route that decodes on the CPU (measured 2026-11-05, shipped binary, Qwen3-0.6B, `-ngl 0`): I srv load_model: KV cache exports its region: a shared prompt handoff moves no bytes E llama_init_from_model: failed to initialize the context: shared KV: CPU allocation or dma-buf export failed E srv llama_server: exiting due to model loading error 1. `share_proc()` fell back to "ask every registered backend" whenever the buffer type had no device -- added so the CPU could be the *importing* side. That fallback must never resolve the *export* proc: no CPU backend exports, so a CPU buffer was handed whichever backend does implement it (ggml-hrx: `libggml-hrx.so` is the only shared object with one, `libggml-cpu.so` has none) and it was called on a foreign buffer. It is now restricted to `ggml_backend_dev_buffer_from_dmabuf`, whose caller passes the buffer type's own device (null for the CPU, which ggml-cpu's importer accepts). 2. `llama_share_alloc()` threw when the export proc was missing or an allocation/export failed. It now warns once with the backend's own name, closes any fds it had already exported, clears the region list and returns an ordinary `ggml_backend_alloc_ctx_tensors_from_buft` buffer. The context is then created normally, `llama_kv_share_regions()` reports 0 regions, and `?action=share` refuses that context per request -- which is a state a caller can act on (the engine's hybrid route falls back to its state-file handoff for that request) instead of a backend that never came up. The server's refusal says both causes now, instead of claiming the flag was not set: 501 {"error":{"message":"This server's memory has no shareable region: either it was not created with ONEBIT_KV_SHARE=1, or its KV backend cannot export memory (the load-time warning names the backend)"}} Measured on the same binary, before and after, no device (release build of this branch, `-ngl 0`, `--parallel 1`): | | before | after | |---|---|---| | flag on, load | process exits: `shared KV: CPU allocation or dma-buf export failed` | `/health` 200, one warning: `share_degrade: CPU cannot export its memory; its KV cache is allocated normally and a shared handoff will be refused for this context (the state-file handoff still works)` | | `?action=share` | unreachable (server dead) | 501 with the message above | | decode | unreachable | `/completion` → ` Paris. The capital of France is also` (8 tokens) | And through the engine's hybrid route with the flag on, one request against this decoder and a real model: the route logs `share answered 501: ...; this decoder cannot share its KV, so the state-file route is used` and then `15 prompt tokens: prefill 20 ms, restore 0 ms, share 0.000 ms, first byte 160 ms`; the reply reports `cached_tokens 14` of 15, i.e. the decoder evaluated the last token only (engine ggml-org#941's contract) on the fallback path. A decoder whose backend CAN export is untouched: the export proc still resolves through its own registry. Refs engine#1203 (the per-request fallback this composes with) and engine#753. Co-authored-by: agent <agent@local>
…kernel (engine ggml-org#991) (#213) Zamba-7B-v1's attention has 464-wide heads (29 x 16) and its 13 FLASH_ATTN_EXT nodes were the only nodes left off HRX: `13 of 5048 nodes off (FLASH_ATTN_EXT x 13)`, 27 splits, and `GGML_HRX_STRICT=1` aborts on `fattn-2`. 464 is not a multiple of 64, which is what the decode-split kernel declared on `value_head_size` (`range(64, 512), multiple_of(64)`), and the HRX matcher mirrored that declaration. The kernel does not need a 64-wide tile: it publishes a head through `ceil(value_head_size / 128)` output tiles, stores 4-wide vectors at `output_tile * 128 + lane_output_channel`, and clamps every one of those stores with `lane_output_channel < value_head_size`. Its real requirements are a 16-value WMMA fragment (`value_head_size % 16`) and a whole 4-wide channel group (`% 4`, which 16 implies). The 64 block belongs to the *prefill* op, which divides by it (`output_tile_count = value_head_size / 64`) and publishes whole 64-wide tiles with no tail path. So this widens the decode side only: - `flash_attention_decode_split_f32_f16_wmma.loom`: `multiple_of(value_head_size, 64)` -> `16` in the config declaration and in all seven `index.assume`s that repeat it. - `dispatch-flash-attention.cpp`: `kDecodeValueHeadSizeBlock = 16` and `is_supported_decode_value_head_size()`, used by the decode-split matcher alone; the prefill matcher keeps `kPrefillValueHeadSizeBlock = 64`. - The decode predicate's upper bound becomes `kDecodeMaxValueHeadSize = 512`, the ops' own declared value range, instead of the wider Q/K cap (576): a contraction the JIT would reject must never be claimed, because a rejected claim is a hard `failed to prepare command` while declining is a CPU fallback. No test case is added: `test-backend-ops` cannot reach this matcher. Its `FLASH_ATTN_EXT` cases at 464 (and at the old widths 64/192) are all refused by `match_decode_split_flash_attention_f32_f16` at its `has_query_layout` gate, which requires the permuted `[tokens][heads][head_size]` layout a model graph produces while the suite builds contiguous `[head_size, tokens, heads]` tensors; the suite cases that do pass at 96/192 are claimed by a different registration (`common.attention_strided_f32_f16`, V % 32). A temporary probe above that gate printed `dsplit-refused: stride-layout` for hsv=464/96/80/192 in `support` mode, so the device evidence for this change is model-level (KLD against the same nodes on the CPU), not a suite result. Nothing else changes: `qk_head_size` keeps its 16 block, the decode matcher's other gates (query length < 16, head counts, KV capacity, the 4096-byte partial alignment) are untouched, and no instruction in any kernel changed. Co-authored-by: agent <agent@local>
llama_adapter_lora_set_factors() copies new lora_a/lora_b values into the tensors the adapter already owns in the backend buffer, through ggml_backend_tensor_set(). The adapter file is not read, no tensor is allocated, and the adapter set on the context does not change, so the next decode reuses the graph. The bulk variant checks the whole batch before writing anything, so a bad entry leaves every weight as it was. n_factors() and factor_name_by_index() let a caller enumerate the weights and rewrite all of them. test-lora-inplace builds a tiny llama model in memory and a rank-2 adapter GGUF, then deletes the adapter file. It checks that the in-place update changes the logits, matches a fresh load of the same factors, restores bit-identical logits when the old values are written back, and that a refused update keeps the old values and the old logits. The factor tensors keep their addresses throughout, so nothing was reloaded. Refs engine#973. Co-authored-by: agent <agent@local>
llama : adapter factors update in place, with no GGUF parse (engine ggml-org#973)
…ph skip metadata views (engine ggml-org#681) A qwen4exp MTP draft head has no tok_embd/output and needs its target context; the memory-fitting probe has none, so the context constructor threw 'requires ctx_other to be set' before the real draft context (which sets ctx_other in common_speculative_init) was ever built. Warn instead of throw: fitting measures, the real draft context still gets ctx_other. The sampler backend support probe rejected the top-k candidate graph on HRX0 because it contains RESHAPE, a metadata-only view the scheduler resolves; skip view ops in the probe. Assisted-by: DeepSeek Harness
A device that supports every host buffer type is the highest-priority backend for host-resident weights, so the scheduler places host-weight nodes on HRX and then spreads the device over the graph that follows. At -ngl 0 that ran most of the graph on HRX over host tensors: 180 of 1266 nodes on the CPU, first-token logprob -1.369900 against -1.084810 on the CPU route, and a greedy essay that loops (engine ggml-org#1078). HRX keeps claiming its own device buffer and host arena; another backend's host buffer is claimed only with GGML_HRX_CLAIM_HOST_BUFTS=1, which keeps the in-place host path available when it is asked for. Measured on strixhalo, Qwen3-4B-IQ4_XS, -c 512 -b 32 -ub 32: -dev HRX0 -ngl 0 now decodes exactly like -dev none (max |dlogprob| 0 over the top-8, 1266 of 1266 nodes on the CPU, essay prose again), and -dev HRX0 -ngl 99 placement and pp/tg are unchanged. Assisted-by: DeepSeek Harness
hrx: claim host buffers only when asked, so -ngl 0 stays on the CPU
…e fitting probe (engine ggml-org#681) Assisted-by: DeepSeek Harness
hrx: attach a separate MTP draft head, and let the sampler support probe skip metadata views (engine ggml-org#681)
…'s, and refuse the 32-expert router launch (engine ggml-org#1259) Full suite at 710c7c2, twice: 4268/4592 both runs, the same 324 failures (suite grew from 4374 since cffae5c4). 1) 318 TOPK_MOE cases fail only on selected_experts, an internal non-contiguous VIEW of the ARGSORT output: the HRX read of that view is garbage while CPU has the ids; comparing the routed weights output (the operation result) passes every shape and gating variant. The real graph consumes the ids inside GET_ROWS on device, so this is a comparison-node artefact, named in a waiver line and still compared on every other backend. 2) two TOPK_MOE with_norm=1 cases are the Loom range failure: router_top8_f32 computes experts_per_lane0 = expert_count / wave_size and assumes [1,16], so 32 experts give 0 and the launch is rejected. The issue attributed this to 31 experts, but the JIT config says expert_count=32. The matcher now requires >=64 experts and a multiple of the 64-lane wave (288 = 4*64+32 left a remainder unscanned, measured 1e-6 in the routed weights); our real MoEs are 128/256. Smaller test graphs run the standalone dispatches. 3) 4 DSV4_HC_POST identity=0 cases: hip.hc_post is a private HIP add-on dispatch, not in this fork, and its non-identity path is wrong there (identity=1 passes). Named waiver, tracked separately. Waiver lines are explicit and printed once per class, e.g. 'WAIVER [HRX0] TOPK_MOE: ...'. Assisted-by: DeepSeek Harness
hrx: name the two backend-ops suite failures that are not the backend's, and refuse the router launch the kernel cannot take (engine ggml-org#1259)
…ds it engine ggml-org#681's warning at llama-context.cpp:160 hands `model.arch_name()` - a std::string - to LLAMA_LOG_WARN, which is a printf-style variadic. Clang refuses that outright in C++: src/llama-context.cpp:160:42: error: cannot pass non-trivial object of type 'std::string' to variadic function; expected type from format string was 'char *' [-Wnon-pod-varargs] so the engine's own payload build (HRX toolchain, amdclang++) fails at this file, and with it every `weekly.sh build` on main since ced49c6 was bumped in (ggml-org#1260). GCC's laxer varargs check is why it can pass a suite elsewhere; the behaviour it was meant to produce is not in doubt, only the argument type. One token: `model.arch_name().c_str()`. Verified by rebuilding the exact targets that failed - `cmake --build build/hrx/llama --target llama-server llama-bench llama-perplexity` under the HRX toolchain - which now compiles this file and proceeds with 0 errors where it previously stopped on this line. It is the only instance: every other `arch_name()` use in the tree is string concatenation or add_kv(), which are fine.
Author
|
Superseded before review: #220 already landed this exact change on master as ab79894 (merged 11:22), one token, same line. Closing so there is one fix and one pin bump (engine ggml-org#1267) rather than two routes to the same thing. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
engine #681's warning atllama-context.cpp:160handsmodel.arch_name()— astd::string— toLLAMA_LOG_WARN, whichis a printf-style variadic. Clang refuses that outright in C++:
Consequence upstream of this repo: the engine's own payload build (
weekly.sh build, HRX toolchain,amdclang++) stopsat this file, so every engine
mainsinceced49c61was bumped in (ggml-org#1260) cannot produce a package. That is the currentblocker for the c31 stick.
One token:
model.arch_name().c_str(). It is also the only instance — every otherarch_name()use in the tree is stringconcatenation or
add_kv(), which are fine. The intended behaviour is unchanged; only the argument type.Verified by rebuilding the exact targets that failed, under the same toolchain the engine uses:
(GCC's laxer varargs check is why a suite elsewhere can be green on this commit; the engine's clang is the one that matters
for what we ship.)
Once this is merged I will bump
third_party/llama.cppin the engine to the merge commit — that bump is what unblocks c31.