Repository navigation
Steps 3b-3c: NPU fast lane runtime on full ELFs, served through Lemonade - #9
Merged
Merged
Conversation
3b, the lane runtime: npu/model (q4nx + config.json reader), npu/pack (layer and lm-head buffer packing, i5/i6, RoPE), npu/lane (XRT, init once, per-context configs generated in memory), npu/generate (overlapped decode, repetition penalty, EOS) and `1bit npu-run`. 3c, serving: npu/tokenizer (Qwen3 tokenizer.json, PCRE2), `1bit unified` (health, models, chat and text completions, SSE), and `1bit lemonade` registers NPU model directories under ONEBIT_MODEL_ROOTS with onebit. On Strix Halo with Qwen3-0.6B: logits bit-identical to the reference lane on 24/24 steps, 11.0 ms/token, 380 ms load; Lemonade serves the model on the NPU through onebit at 87 tok/s. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
force-pushed
the
npu-lane
branch
from
September 23, 2026 04:12
35db83c to
f8c3616
Compare
bong-water-water-bong
enabled auto-merge (squash)
September 23, 2026 04:21
bong-water-water-bong
pushed a commit
that referenced
this pull request
Sep 26, 2026
…oints and three HRX fixes Our llama.cpp #7 (HRX MoE router route-ids binding length), #9 (decode-split flash-attention capacity bound), #11 (ZAYA sliding-window attention, ZAYA1-74B), #12 (MoE router experts-per-wave launch geometry) and #14 (ZAYA legacy checkpoints; --remote fetches chat_template.jinja). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 2, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 2, 2026
…device was chosen (#294) Lemonade fork #9. Before it, an unset onebit_backend read back as vulkan, so Lemonade always launched 1bit serve on Vulkan and PrismML ternary / Hadamard-folded files failed to load (found by the release pre-check). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Oct 7, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 20, 2026
…6 on HRX (#476) * Pin gpu-kernels be68256d and llama.cpp b14a2836: census gaps G1, G3-G6 gpu-kernels #6 (q8_0 KV cache: SET_ROWS + flash attention), #7 (LFM2 multi-sequence MUL_MAT), #8 (ZAYA ragged lm_head), #9 (PAD f32); llama.cpp fork #113 (zero-element CPY elided), #114 (PAD params), #115 (Loom rmsnorm_gate footprint). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * registry: regenerated at llama.cpp b14a2836 Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This was referenced Nov 6, 2026
bong-water-water-bong
added a commit
that referenced
this pull request
Nov 6, 2026
…explicit (engine #840, port of fork #9) (#1245) * Bump HRX: hrx-system 7f4236924 -- the APU memory-properties decode is explicit (engine #840, port of fork #9) * hrx-system: record 7f4236924 as the long-prompt-validated pin (hw-gate run 42170492166) --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #8 (Step 3a, merged). Rebased onto main; the diff is only 3b and 3c.
What
3b, the lane runtime (the port of 1bit-MONSTER's
RuntimeLayerEnginefast-lane path, on full ELFs)npu/model: readsmodel.q4nxandconfig.json.npu/pack: packs layer and lm-head buffers,i5/i6, and RoPE rows (θ = 1e6).npu/lane: XRT runtime. It runs the init ELF once and generates each context length's config in memory the first time that length is reached.npu/generate: overlapped decode on two slots, a repetition penalty (1.1 over 64 ids by default) and EOS.1bit npu-run.3c, serving
npu/tokenizer: ported from 1bit-MONSTER'sqwen3_tokenizer.cpp. One fix: added tokens withspecial: false(<think>,</think>) now stay whole, as HFtokenizersdoes.1bit unified -m <model dir> -p <port>:/v1/health,/v1/models,/v1/chat/completions(with SSE) and/v1/completions.1bit lemonaderegisters every NPU model directory underONEBIT_MODEL_ROOTS(default~/models) with theonebitbackend.The NPU lane builds only with
-DONEBIT_NPU=ON(XRT). CI builds the host side and runsnpu_pack_testandtokenizer_test. Hardware tests register when their input paths are set.Checks (Strix Halo, Qwen3-0.6B)
i5,i6vs the reference lane's buffers (layers 0 and 2)tokenizers0.23.1onebit→1bit unifiedon the NPUThe
1bitbinary links onlylibxrt_coreutil. It needs no xclbin, no FastFlowLM and no Python.Step 3 is done when this merges: Lemonade loads Qwen3-0.6B on the NPU through
onebit.🤖 Generated with Claude Code