Repository navigation
DwarfStar (antirez/ds4) as a backend: 1bit serve --device ds4 - #111
Merged
Merged
Conversation
DwarfStar is a native engine for a few large MoE models (DeepSeek V4/V4.1 Flash, GLM 5.x Flash, Qwen3.8-Flash-Next) in its own GGUF layouts, with first-class Strix Halo ROCm support and SSD expert streaming. It runs as a child of 1bit serve, like ZINC. - third_party/ds4 pins antirez/ds4 0aaea5a238fb (MIT); bump-ds4.yml follows upstream main daily. - scripts/build-ds4.sh builds a copy of the tree (the submodule stays clean) for rocm, cuda, metal or cpu. For ROCm it finds TheRock's SDK and passes its include/ with -isystem: clang otherwise searches it after /usr/include, where a distro HIP of another version shadows it. DS4_TEST=1 runs DwarfStar's model-free routed-MoE GPU test. - ONEBIT_DS4 builds it; serve --device ds4 starts ds4-server (--ctx-size, --ssd-streaming, --ds4 PATH). A child now carries its own readiness path: ds4-server has no /health and opens its port once the model is loaded, so serve waits on its /v1/models. - docs/dwarfstar.md; serve.md, README and NOTICE updated. Verified on Strix Halo (TheRock ROCm): builds, routed-MoE test PASS, engine ctest 7/7, the missing-model and --ssd-streaming error paths. A full model (DeepSeek V4 Flash Q2, 81 GiB) is still to run: no room on the box. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Docs7 for 1bit-monster/engine
Commit |
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
bong-water-water-bong
enabled auto-merge (squash)
September 26, 2026 02:25
bong-water-water-bong
added a commit
that referenced
this pull request
Oct 20, 2026
…erver directly (#475) * serve --print-backend: one hop from Lemonade for GGUF models A chat request through Lemonade went lemond -> 1bit serve -> llama-server: two HTTP hops and three processes. `1bit serve ... --print-backend` makes every check serve makes before a load (architecture, memory admission, placement flags, recipes, the HRX environment) and prints the llama-server it would start, on the port the caller passed, as one JSON line, then exits: 0 printed, 3 does not fit, 5 the route needs serve in the request path (NPU directory or --device npu, onnx, ds4, --pm, --laya, --embed/--rerank). Lemonade's onebit recipe (fork branch 1bit/one-hop) starts that llama-server itself and forwards to it directly. The plan adds --alias (Lemonade's name) and, for chat models, --cache-reuse 256 unless a recipe set it; env lists only variables the caller's environment does not already set, so a user value still wins as under serve. What serve did on the request path moves into the backend: - llama.cpp pin 680933de1 (fork branch 1bit/one-hop, on #111): /health answers 503 while the main loop is stalled with work queued, so Lemonade's BackendWatchdog probe resets a hung GPU, and llama-server exits on a lost GPU (serve's #466 signal); - --strict-device stays GGML_HRX_STRICT in the backend. third_party/lemonade pins the fork's 1bit/one-hop commit 23972095. registry/architectures.json regenerated for the new llama.cpp pin. Test: tests/print_backend.sh (ctest print_backend, no GPU): the JSON plan, port, alias, --cache-reuse, ctx, the embedding role, exit 5 for NPU/--pm/--embed, exit 3 for a model that does not fit, nothing started. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * docs/serve.md: one hop measured on Strix Halo; tools/onehop_bench.py tools/onehop_bench.py starts a private lemond (own cache/config dirs and port), loads one onebit model on HRX and measures load time, /tokenize and a 1-token chat via Lemonade and direct, time to first token, and (one-hop arm) a pinned reload beside a second model and an unload + load. Qwen3-4B-Instruct-2507 Q4_K_M on hrx, six rounds per arm, same llama-server for both: the serve hop cost about 0.2 ms a request (/tokenize 0.53-0.64 ms -> 0.36-0.39 ms via Lemonade, 0.11-0.15 ms direct); load 1.62 s -> 1.35-1.56 s; chat and time to first token unchanged within noise. A pinned reload keeps the same llama-server and its warm program cache (seen length 106-119 ms); an unload + load starts cold (228-236 ms). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * Pin llama.cpp 3c24f910f (fork #112) and lemonade 80064ba7 (fork #15): the squash merges of the one-hop branches Same trees as 680933de1 and 23972095, which the hw-gate run on this PR tested. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> * registry: regenerated at llama.cpp 3c24f910f Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> --------- Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
DwarfStar (MIT) is a native engine for a few large MoE models (DeepSeek V4/V4.1 Flash, GLM 5.x Flash, Qwen3.8-Flash-Next) in its own GGUF layouts, with first-class Strix Halo ROCm support and SSD expert streaming. Its GGUFs are the most downloaded DeepSeek V4.1 Flash files on HF (
antirez/deepseek-v4.1-flash-gguf, 940k), and llama.cpp cannot load them. This adds it as a child backend, like ZINC.third_party/ds4: antirez/ds40aaea5a238fb;bump-ds4.ymlfollows upstream main daily.scripts/build-ds4.sh: builds a copy of the tree (the submodule stays clean) for rocm / cuda / metal / cpu. ROCm finds TheRock's SDK and passes itsinclude/with-isystem(clang otherwise searches it after/usr/include, where a distro HIP of another version shadows it).DS4_TEST=1runs DwarfStar's model-free routed-MoE GPU test.-DONEBIT_DS4=ON;1bit serve --device ds4with--ctx-size,--ssd-streaming,--ds4 PATH. A child now carries its own readiness path:ds4-serverhas no/healthand opens its port once the model is loaded, soservewaits on/v1/models.docs/dwarfstar.md; serve.md, README (repo XDNA: pin upstream amd/xdna-driver (and its XRT), build it privately, keep it current #12) and NOTICE.Verified on Strix Halo (TheRock ROCm):
DS4_TEST=1 scripts/build-ds4.sh build/ds4 rocm-DONEBIT_DS4=ONserve --device ds4 -m <missing>--ssd-streamingon another deviceNot yet run: a full model. DeepSeek V4 Flash Q2 is 81 GiB on disk and needs about that much free memory; the box has 89 GB of disk and ~60 GB of memory free while other work runs. Hold the merge until it answers a chat through
serve --device ds4.🤖 Generated with Claude Code