GGUF-on-NPU: serve all six target arches on --device npu by bong-water-water-bong · Pull Request #185 · 1bit-MONSTER/engine

bong-water-water-bong · 2026-09-28T05:44:21Z

Summary

1bit serve -m <model.gguf> --device npu now repacks a GGUF into the Q4NX int4
tile format and runs the NPU lane end-to-end for six target architectures. Each
answers the anchor prompt ("The capital of France is", greedy, max_tokens=1):

arch	model	answer
qwen2	Qwen2.5-7B-Instruct Q4_K_M	" Paris"
qwen3moe	Qwen3-Coder-30B-A3B Q4_K_M	" Paris"
qwen35moe	Qwen3.6-35B-A3B Q8_0	" Paris"
deepseek2	GLM-4.7-Flash Q4_K_M	" Paris"
minicpm	MiniCPM4-8B Q4_K_M	"Paris"
llama	MiniCPM5-1B Q4_K_M	" Paris"

Captured runs: evidence/gguf-npu-six/rerun_*.json + task4_READY.md
(goal muhz0s39-8c9l9o). The vulkan and hrx lanes are unchanged (serve_e2e
PASS), and the existing Q4NX-directory fast lane stays bit-identical on its
golden check (24/24).

Changes

deepseek2 / GLM-4.7-Flash — the converter's k_b per-head transpose is
inverted; the engine repack disables that gate in its patched converter copy
(analysis below), and the two missing GLM MLA fixes are resynced (expert
transpose, v_b grid).
qwen3moe — per-expert MoE in the standard path.
qwen35moe — GDN + MoE hybrid (Qwen3.6-35B-A3B): GDN geometry +
attn_output_gate, the hybrid forward, and six new kernels.
NPU serialisation — 1bit serve takes /tmp/1bit-npu-device.lock like
the ELF runner, so concurrent serves don't fight over the one device.
Tokenizer — leading-space decode fix + Unigram tokenizer.json accepted
(MiniCPM4).
Repack provenance — the repack carries the GGUF arch so the
provenance-stamp check re-resolves the converter.
Registry — the six npu GGUF rows (check_models.tsv + registry_check.py
npu-.gguf support) and the anchor re-run harness (tests/anchor_rerun.sh).

Verification

Six captured --device npu runs: evidence/gguf-npu-six/rerun_{q2,q3m,q35,glm,m4,m5}.json.
tests/serve_e2e.sh vulkan PASS and hrx PASS (MiniCPM5-1B).
tests/npu_lane_e2e.sh 24/24 bit-identical (Qwen3-0.6B).

GLM k_b transpose (original analysis)

FLM_Q4NX_Converter commit 62594563f ("transpose k_b's per-head slice to the
MLA math grid") is inverted. Measured with the converter's own _pack_2d_float
on GLM's blk.0.attn_k_b.weight:

per-head slice	packed	tiles/head
`[512][192]` (kv_lora x nope)	`[12][5120]`	12
`[192][512]` (after transpose)	`[16][5120]`	16

The forward reads k_b_proj wanting the 12-tile grid (its own comment says so),
and the pre-62594563f Q4NX the GLM capture was verified against has
k_b [20][12][5120]. With the transpose in place the repack emitted
k_b [20][16][5120], the forward read past the slice, and the serve segfaulted
on the first request. The engine repack now runs the converter from a patched
copy with that gate disabled (the same way it already forces Q4_1) and prints
why; the converter repo is untouched.

Performance (measured 2026-09-28, `--device npu`)

8-token greedy continuation of "The capital of France is" (load = serve start → /health 200):

model	load	8-token gen	tok/s
MiniCPM5-1B (llama)	19.0 s	47.0 s	0.170
Qwen2.5-7B (qwen2)	73.6 s	434.0 s	0.018
Qwen3-Coder-30B-A3B (qwen3moe)	4.0 s	488.0 s	0.016
GLM-4.7-Flash (deepseek2)	604.8 s	636.6 s	0.013
MiniCPM4-8B (minicpm)	112.2 s	1130.9 s	0.007
Qwen3.6-35B-A3B (qwen3_5_moe)	6.5 s	1416.8 s	0.006

Correctness across all six arches is the milestone. Decode is slow (0.006–0.17 tok/s)
because this is the model-generic forward (~100+ synchronous NPU dispatches/token,
correctness-first), not the fused fast lane (~96 tok/s on Qwen3-0.6B). Full run
details: evidence/gguf-npu-six/benchmark-2026-09-28.md.

The converter's 'k_b to the MLA math grid' transpose (FLM_Q4NX_Converter 62594563f) is inverted. Measured directly with the converter's own _pack_2d_float on GLM's blk.0.attn_k_b.weight: the dequantised per-head slice is [512][192] (kv_lora x nope) and packing it AS-IS gives the 12 tiles/head the forward reads (load_heads3 with kb_rows_ = qk_nope_ = 192, cols = kv_lora_ = 512), while transposing it to [192][512] gives 16 tiles/head. The forward then reads past the slice and '1bit serve --device npu' segfaults on the first request. A repack via scripts/repack_gguf.py therefore emits k_b [20][16][5120] while the Q4NX the GLM capture was verified against (/tmp/glm_v3) has [20][12][5120]. The engine's repack now runs the converter from a patched copy with that gate disabled, the same way it already forces Q4_1, and prints why. Verified: the patched repack writes k_b [20][12][5120] (v_b unchanged at [20][16][5120]).

context7 · 2026-09-28T05:44:30Z

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit b6b5864

github-actions · 2026-09-28T05:44:56Z

PR Reviewer Guide 🔍

(Review updated until commit `b6b5864`)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review NPU Device Locking The NPU device locking mechanism uses a global lock file `/tmp/1bit-npu-device.lock` and `flock` to serialize access. While this approach prevents concurrent `1bit serve` processes from failing with EINVAL, it introduces a potential race condition where a process might be stuck waiting for the lock indefinitely if another process dies without releasing it. Additionally, the lock is held for the lifetime of the process, which could lead to long-term blocking if the process is not cleanly terminated. if (!std::getenv("NPU_NO_DEVICE_LOCK")) { const char* lk = std::getenv("NPU_DEVICE_LOCK"); if (!lk \|\| !lk[0]) lk = "/tmp/1bit-npu-device.lock"; int lfd = ::open(lk, O_CREAT \| O_RDWR, 0666); if (lfd >= 0) { if (::flock(lfd, LOCK_EX \| LOCK_NB) != 0) { std::fprintf(stderr, "[npu] waiting for the exclusive device lock (%s)...\n", lk); if (::flock(lfd, LOCK_EX) != 0) std::fprintf(stderr, "[npu] device lock unavailable; continuing unlocked\n"); else std::fprintf(stderr, "[npu] device lock acquired\n"); } } } MLA Expert Transpose The change in `mla_expert` function removes the explicit transpose operation that was previously used to convert the flat tensor order into the logical matrix order. This change is critical for correctness, as it was verified to produce the correct correlation with the GGUF tensor order. If this change is not carefully validated across all affected architectures, it could lead to incorrect model outputs. // Do NOT transpose. The tile decode at (rows=K, cols=N) already reproduces the // GGUF's flat order, and that flat sequence IS the logical [N][K] = [out][in] // weight. Verified elementwise against the GGUF per expert: reshape(N,K) gives // corr 0.999198 (gate) / 0.996949 (down), while transposing gives -0.001079 / // 0.000417 -- i.e. the transpose destroyed the expert entirely, making every // routed expert's contribution ~10-20x wrong and flipping the top-k selection. std::copy(kn.begin(), kn.end(), out.begin()); return true; MoE Layer Loading The MoE layer loading logic has been updated to correctly parse and load MoE parameters outside of the MLA block. This change ensures that non-MLA MoE architectures like qwen3moe and qwen35moe are properly handled. However, the logic for loading shared experts and router weights needs to be carefully validated to ensure that all MoE configurations are correctly supported. if (n_expert_ > 0) { // Qwen MoE layer (qwen3moe / qwen35moe): a router + top-k routed experts // instead of a dense gate/up/down. The router is bf16; its bias is OPTIONAL // (Qwen MoE has none, deepseek2 does). The experts stay pooled -- one // [N][K] slice is dequantized per selected expert at decode. char nm2[160]; auto td2 = [&](const char* fmt) -> TensorDesc* { snprintf(nm2, sizeof(nm2), fmt, l); return model_tensor_by_name(mw_, nm2); }; auto load_bf16_opt = [&](const char* fmt, std::vector<float>& v, int n) -> bool { TensorDesc* d = td2(fmt); if (!d) return false; const uint16_t* p = (const uint16_t)model_tensor_data(mw_, d); if (!p) return false; v.resize((size_t)n); for (int i = 0; i < n; i++) v[i] = bf16_to_f32(p[i]); return true; }; if (!load_bf16_opt("model.layers.%d.moe_router.weight", moe_router_, n_expert_ H)) { err_ = "moe router"; return false; } if (!load_bf16_opt("model.layers.%d.moe_router.bias", moe_router_b_, n_expert_)) moe_router_b_.clear(); auto pool = [&](const char* fmt, const uint8_t** p, size_t* bytes) -> bool { TensorDesc* d = td2(fmt); if (!d) { fprintf(stderr, " [moe] missing %s\n", nm2); return false; } const uint8_t* raw = (const uint8_t)model_tensor_data(mw_, d); if (!raw) return false; p = raw; bytes = (size_t)d->data_size / (size_t)n_expert_; return true; }; if (!pool("model.layers.%d.mlp.gate_exps_proj.weight", &exp_g_, &exp_g_bytes_)) return false; if (!pool("model.layers.%d.mlp.up_exps_proj.weight", &exp_u_, &exp_u_bytes_)) return false; if (!pool("model.layers.%d.mlp.down_exps_proj.weight", &exp_d_, &exp_d_bytes_)) return false; // Optional shared expert (qwen35moe): a 2D [expert_im][H] / [H][expert_im] pair // plus its [H] gate. Cleared when absent (qwen3moe has none). auto load_share = [&](const char fmt, std::vector<float>& v, int rows, int cols) -> bool { TensorDesc* d = td2(fmt); if (!d) return false; v.resize((size_t)rows * (size_t)cols); const uint8_t* raw = (const uint8_t)model_tensor_data(mw_, d); return raw && dequant_any(raw, (size_t)d->data_size, rows, cols, v.data()) == 0; }; if (!load_share("model.layers.%d.mlp.share_gate_exps_proj.weight", share_g_, expert_im_, H)) share_g_.clear(); if (!load_share("model.layers.%d.mlp.share_up_exps_proj.weight", share_u_, expert_im_, H)) { share_g_.clear(); share_u_.clear(); } if (!load_share("model.layers.%d.mlp.share_down_exps_proj.weight", share_d_, H, expert_im_)) { share_g_.clear(); share_u_.clear(); } if (!load_bf16_opt("model.layers.%d.mlp.share_router.weight", share_router_, H)) share_router_.clear(); return true; } GDN Layer MoE Handling* The GDN layer MoE handling mirrors the standard MoE loading logic. This ensures that GDN layers with MoE components are correctly loaded and executed. However, the correctness of this implementation depends on the proper handling of the MoE parameters and the integration with the common MLP section. if (n_expert_ > 0) { // qwen35moe: a GDN layer's FFN is MoE, not a dense gate/up/down. The // loading mirrors the standard path's MoE branch; the FFN itself is run by // the common MLP section (which sees n_expert_ > 0 and calls moe_mlp). char nm2[160]; auto td2 = [&](const char* fmt) -> TensorDesc* { snprintf(nm2, sizeof(nm2), fmt, l); return model_tensor_by_name(mw_, nm2); }; auto load_bf16_opt = [&](const char* fmt, std::vector<float>& v, int n) -> bool { TensorDesc* d = td2(fmt); if (!d) return false; const uint16_t* p = (const uint16_t)model_tensor_data(mw_, d); if (!p) return false; v.resize((size_t)n); for (int i = 0; i < n; i++) v[i] = bf16_to_f32(p[i]); return true; }; if (!load_bf16_opt("model.layers.%d.moe_router.weight", moe_router_, n_expert_ H)) { err_ = "moe router"; return false; } if (!load_bf16_opt("model.layers.%d.moe_router.bias", moe_router_b_, n_expert_)) moe_router_b_.clear(); auto pool = [&](const char* fmt, const uint8_t** p, size_t* bytes) -> bool { TensorDesc* d = td2(fmt); if (!d) { fprintf(stderr, " [moe] missing %s\n", nm2); return false; } const uint8_t* raw = (const uint8_t)model_tensor_data(mw_, d); if (!raw) return false; p = raw; bytes = (size_t)d->data_size / (size_t)n_expert_; return true; }; if (!pool("model.layers.%d.mlp.gate_exps_proj.weight", &exp_g_, &exp_g_bytes_)) return false; if (!pool("model.layers.%d.mlp.up_exps_proj.weight", &exp_u_, &exp_u_bytes_)) return false; if (!pool("model.layers.%d.mlp.down_exps_proj.weight", &exp_d_, &exp_d_bytes_)) return false; // Optional shared expert (qwen35moe): a 2D [expert_im][H] / [H][expert_im] pair // plus its [H] gate. Cleared when absent. auto load_share = [&](const char fmt, std::vector<float>& v, int rows, int cols) -> bool { TensorDesc* d = td2(fmt); if (!d) return false; v.resize((size_t)rows * (size_t)cols); const uint8_t* raw = (const uint8_t)model_tensor_data(mw_, d); return raw && dequant_any(raw, (size_t)d->data_size, rows, cols, v.data()) == 0; }; if (!load_share("model.layers.%d.mlp.share_gate_exps_proj.weight", share_g_, expert_im_, H)) share_g_.clear(); if (!load_share("model.layers.%d.mlp.share_up_exps_proj.weight", share_u_, expert_im_, H)) { share_g_.clear(); share_u_.clear(); } if (!load_share("model.layers.%d.mlp.share_down_exps_proj.weight", share_d_, H, expert_im_)) { share_g_.clear(); share_u_.clear(); } if (!load_bf16_opt("model.layers.%d.mlp.share_router.weight", share_router_, H)) share_router_.clear(); return true; } MoE MLP Implementation* The MoE MLP implementation now includes logic to handle the shared expert with sigmoid gating. This is important for architectures like qwen35moe, where the shared expert is gated using a sigmoid function. The implementation needs to be validated to ensure that the gating mechanism is correctly applied and that the shared expert contribution is properly integrated. // Qwen MoE (qwen35moe) gates the shared expert with sigmoid(x . g), a [H] // vector; deepseek2 has no such gate (share_router_ empty -> scale 1). float sgate = 1.0f; if (!share_router_.empty()) { float g = 0.0f; for (int i = 0; i < H; i++) g += share_router_[i] * xn[i]; sgate = 1.0f / (1.0f + std::exp(-g)); } for (int i = 0; i < H; i++) d[i] += sgate * sd[i]; LM Head GEMM Design Selection The LM head GEMM design selection logic has been updated to handle cases where the QKV design is not available for certain architectures like GLM-4.7-Flash. This prevents a SIGFPE error that could occur when the design selection fails. However, the robustness of this fallback mechanism needs to be validated across all supported architectures. // Reuse an existing GEMM geometry for the head (the variable keeps its old name // for the block below). QKV when the arch has one; the MLA/deepseek2 arch // (GLM-4.7-Flash) has QA/QB/KVA/O instead, so idx_of(QKV) returned -1 and T was // read out of designs_[-1] -- garbage, and the ntiles division below raised // SIGFPE. Only this NPU path uses T (the host path is a plain matvec), which is // why GLM crashed here the first time the ELF path was exercised. int i_qkv = idx_of(Q4NX_DESIGN_QKV); if (i_qkv < 0 \|\| designs_[i_qkv].K != H \|\| designs_[i_qkv].N <= 0) i_qkv = idx_of(Q4NX_DESIGN_DG); // K=H, N=dense_im if (i_qkv < 0 \|\| designs_[i_qkv].K != H \|\| designs_[i_qkv].N <= 0) i_qkv = idx_of(Q4NX_DESIGN_G); // K=H, N=expert_im if (i_qkv < 0 \|\| designs_[i_qkv].K != H \|\| designs_[i_qkv].N <= 0) { err_ = "LM head: no GEMM design with K=H available"; return false; } const int T = designs_[i_qkv].N;

bong-water-water-bong · 2026-09-28T05:59:08Z

Review: please fix this at the source rather than in the engine. repack_gguf.py now detects the converter's k_b transpose by matching a line of its source text and turns it off in a patched copy. Two problems:

Fragile. Any rewording of that line in the converter silently turns the workaround off, or leaves it on after the converter is fixed, and the result segfaults again (or reads wrong tiles) without an error.
Two writers disagree. The converter's own author added that transpose for the MLA grid; this PR says it's inverted. Only one of them can be right, and the fix belongs in the converter, plus a check the forward can make (e.g. assert k_b is 12 tiles per head, kb_rows_ == qk_nope) so a wrong layout fails loudly at load.

Suggest: settle the orientation with the converter's author, fix it there, and drop the patch here. If a stopgap is needed meanwhile, gate it on a converter version or config field, not on source text, and fail loudly when neither form matches.

bong-water-water-bong · 2026-09-28T06:58:57Z

The converter now reverts the inverted k_b transpose at the source (5fc52149b, "converter deepseek2: revert the inverted k_b transpose (62594563f)"). This PR's source-text patch can be dropped: close it, or reduce it to a load-time check in the forward that k_b is 12 tiles per head.

…grid) The MLA/MoE path in this re-synced forward was missing two canonical npu-infer fixes, so `1bit serve -m GLM-4.7-Flash-Q4_K_M.gguf --device npu` decoded " jp" instead of " Paris": 1. mla_expert() transposed every routed expert. Its tile decode already reproduces the GGUF's flat order, which IS the logical [N][K] weight, so the transpose made every routed expert ~10-20x wrong and flipped the top-k selection (canonical f336dfc72). Now a plain std::copy. 2. v_b was indexed in-major (wv[k*vh+n]); the Q4NX's logical index is out-major (h*256+n)*512+k (canonical 3d1b252ec). Now wv[n*kv_lora+k]. Verified on the NPU: /v1/completions "The capital of France is" -> " Paris" (4m4s). Q4NX is byte-identical to the verified glm_npu2 dir; the two bugs were the only functional deltas from canonical.

…-device npu) Qwen3-Coder-30B-A3B (qwen3moe) is the first NON-MLA MoE arch: the FFN is a 128-expert top-8 softmax router with no shared expert and no router bias. Four changes make `1bit serve -m <gguf> --device npu` answer " Paris": 1. forward: parse the MoE params (n_expert/top_k/expert_im/dense_im/ leading_dense/gating/scale) INDEPENDENTLY of mla_, so a non-MLA MoE arch gets its expert geometry (they were read only inside `if (mla_)`). 2. forward: load the MoE tensors (bf16 router + pooled gate/up/down experts) in the standard path when n_expert_>0, instead of the dense gate/up/down; the runner bias and shared expert stay optional (moe_mlp already handles both). 3. forward: dispatch moe_mlp in the standard path for n_expert_>0 (deepseek2 still reaches moe_mlp from the MLA branch, which continues first). 4. forward: `dense_weights_cached` must NOT skip load_layer for a MoE arch -- moe_router_ is a HOST vector reloaded per layer, so skipping it left every layer using the last-loaded layer's router on tokens >= 1 (pos 0 was fine, which is why it looked plausible but decoded "."). scripts/repack_gguf.py: write the MoE config keys for qwen3moe/qwen35moe -- the per-EXPERT intermediate (768) must replace the dense feed_forward_length (5472) so the G/U/D designs ARE the expert MLP, plus n_expert/top_k/softmax gating. Verified: anchor -> " Paris" (3m54s); experts/q_proj/o_proj/router all decode corr >= 0.9989 vs the GGUF; GLM (deepseek2) still answers " Paris" (regression).

The device exposes one NPU hardware context; a second concurrent '1bit serve --device npu' dies inside DRM_IOCTL_AMDXDNA_CREATE_HWCTX (EINVAL) and exits 1. npu_engine_universal has always held this flock; the serve route did not, so siblings could not serialise (and my qwen3moe run hit exactly this when a leftover GLM serve held the context).

Qwen3.6-35B-A3B is a hybrid: 30 GDN (gated delta-net linear attention) layers interleaved with 10 full-attention layers, each FFN a 256-expert top-8 softmax router plus a gated shared expert. Extends the qwen3moe work: - forward: load_gdn_layer now takes the MoE branch (router + pooled experts + the optional shared expert) when n_expert_ > 0, instead of the dense gate/up/down; the GDN layers' FFN was already dispatched by the common MLP section (it sees n_expert_ > 0 and calls moe_mlp). - forward: the shared expert (share_gate/up/down_exps_proj) and its [H] gate (share_router) load in both MoE branches; moe_mlp applies sigmoid(x . g) to the shared expert output (qwen35moe), and leaves it unscaled when absent (deepseek2 has no such gate). - repack: write the GDN geometry (linear_num_key_heads/key_head_dim/ value_head_dim/num_value_heads/conv_kernel_dim from the GGUF ssm.* fields), attn_output_gate (the q_proj emits q|gate) and the per-layer layer_types list the forward dispatches on. Kernels generated for the six new geometries (QKV K2048 N9216 for the gated full-attn, G/U K2048 N512, D K512 N2048, QKV K2048 N8192 (GDN qkv), Z K2048 N4096). Q4NX repacked (21 GB) + the GDN config keys verified.

The stamp records converter_path/commit; --check-stamp re-resolves the converter from the dir's config. It used model_type, which for qwen35moe is the HF name 'qwen3_5_moe' (no q4nx/models/qwen3_5_moe.py), so it fell back to the canonical checkout while the dir was built by the iso-build one -> always 'stale' -> the serve discarded and re-repacked the 21 GB dir on every start (and twice hit ENOSPC doing it). config.json now carries "arch" (the GGUF architecture) and the check prefers it.

…er.json Two defects on the SentencePiece path (MiniCPM4-8B, the only SentencePiece model of the four blocked targets), both found by testing against oracle ids rather than by reading the code. 1. Every SPM decode gained a spurious leading space. decode() mapped every metaspace to ' ', including the one spm() synthesises as SentencePiece's add_dummy_prefix marker, so 'The capital of France is' -> ' The capital of France is' and 'Paris' -> ' Paris'. Strip exactly that one leading space: stripping one rather than all also preserves a genuinely space-leading input, since ' Paris' encodes to the double metaspace [1345, 3837, 1360] and must decode back to ' Paris'. 2. A correctly-typed tokenizer.json could not be loaded at all. model.at("merges") throws [json.exception.out_of_range.403] when the key is absent, and a SentencePiece/Unigram tokenizer.json has no such key -- the throw aborted the constructor before the SPM detection, so the unigram path was unreachable and every SPM model fell back to raw bytes. Also accept the Unigram list-form vocab ([[token, score], ...], id = position) as well as the BPE object form. Verified with a negative control: the old expression throws on a Unigram file, the new one returns an empty merge set. Test: tests/data/minicpm4_tokenizer_golden.json, whose ids come from llama.cpp's own tokenizer (llama-server /tokenize on MiniCPM4-8B-Q4_K_M.gguf), not from this tokenizer. Results: MiniCPM4 as repacked 5/5 (was 0/3), MiniCPM4 as a proper Unigram file 5/5 (previously unloadable), Qwen3 BPE golden 14/14 unchanged.

github-actions · 2026-09-28T17:09:40Z

Persistent review updated to latest commit aa9b7f9

+    if (!std::getenv("NPU_NO_DEVICE_LOCK")) {
+        const char* lk = std::getenv("NPU_DEVICE_LOCK");
+        if (!lk || !lk[0]) lk = "/tmp/1bit-npu-device.lock";
+        int lfd = ::open(lk, O_CREAT | O_RDWR, 0666);


+    if (!std::getenv("NPU_NO_DEVICE_LOCK")) {
+        const char* lk = std::getenv("NPU_DEVICE_LOCK");
+        if (!lk || !lk[0]) lk = "/tmp/1bit-npu-device.lock";
+        int lfd = ::open(lk, O_CREAT | O_RDWR, 0666);


- tools/registry_check.py: an npu .gguf row resolves under --models and reads its arch from the GGUF header (the GGUF-on-NPU route repacks it itself), instead of being treated as a Q4NX model directory. - registry/check_models.tsv: add the six npu GGUF rows (qwen2, qwen3moe, qwen35moe, deepseek2/glm, minicpm, llama). - registry/checked.json: refresh the Qwen3-0.6B npu-directory entry to the current HEAD. - tests/anchor_rerun.sh: harness to re-run the anchor completion across a set of served models and capture the results.

github-actions · 2026-09-28T17:25:55Z

Persistent review updated to latest commit 4784b06

bong-water-water-bong · 2026-09-28T17:29:12Z

Still not mergeable:

The source-text k_b patch is still in the diff (scripts/repack_gguf.py, _kb_patch matching 'transpose_slice = "k_b_proj" in nm'). The converter fixed this at the source (5fc52149b), so the patch is now dead code that would silently re-flip the layout if the converter reintroduces that line for another reason. Please drop it and keep only a load-time tile check.
Commit authorship: 7 of 8 commits are agent <agent@local>, agent-872bf5@localhost or pi agent. Commits on this repo go in as bong-water-water-bong <bong-water-water-bong@1bit.gg>: pass -c user.name/-c user.email or --author, since the strixhalo global git config is agent@local. A squash merge fixes the author of the merge commit only if the merger sets it.
CodeQL fails.
The feature itself (six arches on --device npu, including qwen3moe and qwen35moe) is welcome; no kernel detail was added in this diff.

The GGUF-on-NPU route serves six arches, each verified by a captured `--device npu` run (goal muhz0s39-8c9l9o). Record them so the registry reflects what the NPU actually carries — text decoders only: - qwen2 (Qwen2.5-7B), qwen3moe (Qwen3-Coder-30B), qwen3_5_moe (Qwen3.6-35B), deepseek2 (GLM-4.7-Flash), minicpm (MiniCPM4-8B), llama (MiniCPM5-1B). Regenerates registry/architectures.json (npu count 1 -> 7).

github-actions · 2026-09-28T17:47:42Z

Persistent review updated to latest commit b6b5864

bong-water-water-bong · 2026-09-28T23:29:08Z

Superseded by #209 (squash-merged as 4c8e861), which carries these commits. This PR can be closed.

bong-water-water-bong · 2026-09-29T02:03:51Z

Closing as superseded: all nine commits landed through #209 (squash 4c8e861), which is why this branch now conflicts. I checked file by file against main (fd5662d): five files are identical, and in the other seven every line only this branch has is an older version that main replaced on purpose:

the tokenizer's leading-space strip (the streamed-decode bug GGUF-on-NPU: all six architectures answer on --device npu (qwen3moe, qwen35moe, GLM fixes), tokenizer and serve fixes #209 fixed);
the k_b source rewrite (now a loud refusal, repack_gguf: refuse the inverted k_b converter instead of rewriting its source #212);
the QKV-design LM head (widest-design head, GGUF-on-NPU: all six architectures answer on --device npu (qwen3moe, qwen35moe, GLM fixes), tokenizer and serve fixes #209);
gap_violations (hardened in registry: harden the gap guard and pin its firing behavior #200);
the NPU device lock (hardened: O_NOFOLLOW, 0644);
an older llama.cpp pin in the registry.
Nothing is lost by closing.

github-actions Bot added the Review effort 3/5 label Sep 28, 2026

agent and others added 6 commits September 28, 2026 07:22

bong-water-water-bong changed the title ~~repack: neutralise the inverted deepseek2 k_b transpose (GLM-4.7-Flash)~~ GGUF-on-NPU: serve all six target arches on --device npu Sep 28, 2026

github-actions Bot added Review effort 5/5 and removed Review effort 3/5 labels Sep 28, 2026

github-advanced-security Bot found potential problems Sep 28, 2026

View reviewed changes

bong-water-water-bong force-pushed the npu-glm-kb-fix branch from aa9b7f9 to 4784b06 Compare September 28, 2026 17:24

github-actions Bot added Review effort 4/5 and removed Review effort 5/5 labels Sep 28, 2026

bong-water-water-bong mentioned this pull request Sep 29, 2026

repack_gguf: refuse the inverted k_b converter instead of rewriting its source #212

Merged

bong-water-water-bong closed this Sep 29, 2026

bong-water-water-bong deleted the npu-glm-kb-fix branch October 23, 2026 11:04

bong-water-water-bong mentioned this pull request Nov 5, 2026

Bump HRX: llama.cpp 817d0815b — the engine gains Zamba2, Zamba v1 and BlackMamba (registry 294 -> 297) #1039

Merged

Navigation Menu

Repository navigation

GGUF-on-NPU: serve all six target arches on --device npu - #185

GGUF-on-NPU: serve all six target arches on --device npu#185
bong-water-water-bong wants to merge 9 commits into
mainfrom
npu-glm-kb-fix

bong-water-water-bong commented Sep 28, 2026 •

edited

Loading

Uh oh!

context7 Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Sep 28, 2026 •

edited

Loading

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

bong-water-water-bong commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Changes

Verification

GLM k_b transpose (original analysis)

Performance (measured 2026-09-28, --device npu)

Uh oh!

context7 Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

github-actions Bot commented Sep 28, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

PR Reviewer Guide 🔍

(Review updated until commit b6b5864)

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

github-actions Bot commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 28, 2026

Uh oh!

bong-water-water-bong commented Sep 29, 2026

Uh oh!

Reviewers

Uh oh!

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

bong-water-water-bong commented Sep 28, 2026 •

edited

Loading

Performance (measured 2026-09-28, `--device npu`)

context7 Bot commented Sep 28, 2026 •

edited

Loading

github-actions Bot commented Sep 28, 2026 •

edited

Loading

(Review updated until commit `b6b5864`)