[bong-water-water-bong]

Design note for the owner ask of 2026-11-03: Lemonade starts with no model loaded, and the first request must be decided by Laya and answered by one of our own models on the hub, on our kernels ("we need to own that space completely hip kernels and our npu engine").

What this doc says

Our fork already routes every request through a policy engine (collection.router, docs/dev/router-policy.md): ordered rules over conditions pick a candidate model and the decision is reported in x-lemonade-route / x_lemonade_route. So the cold first request needs four things, none of which is a new router:

1. a classifier condition backed by Laya — 25 ms on HRX0, instead of loading an LLM to decide;
2. a shipped table naming our 1bit-MONSTER repos, file, device (hrx / npu / hybrid) and ctx per class;
3. the name auto resolving to that table;
4. the scorer warm when the first request arrives (627 ms daemon ready, not 2.7 s of weights per request).

The measured before, on the fork build 2026.46.0~1778.8888dd42 (2026-11-03, 127.0.0.1:13499, empty cache)

probe	result
POST /v1/routing/validate, deterministic policy, three prompts	code → ZAYA1-8B in 9 ms, a 2,170-char document → Qwen3.8-27B in 4 ms, a one-liner → default in 3 ms, matched_rule/default_used correct
POST /v1/chat/completions with "model": "auto"	{"error":{"code":"model_not_found", …}}
validate with a {"type":"laya"} classifier	{"error":"Invalid routing policy: unknown classifier type: laya"}

The routing engine, decision format and validator work today with zero new code; the two gaps are exactly Laya-as-a-condition and auto-as-a-name.

Contents

Request flow with the fork's own symbols (Router::resolve_model_name router.cpp:160, try_get_model_info :164, routing engine routing_policy.cpp/routing_policy_parser.cpp:417-440, Router::load_model :849, OnebitServer::load onebit_server.cpp:220 and the 1bit serve argv at :242-250, the ClassifierServices seam at routing_policy.h:162); the routing-table JSON with our real Hub repos and file names (verified against the Hub API: 68 1bit-MONSTER repos, all four classes have a first-party repo with the file named); cold-start behaviour including preload, the 25 ms/627 ms/2.7 s decision costs, pull-on-miss with a budget and a marked degraded fallback; the override rule (an explicit model always wins, Laya failure is Score.ok=false → match_false → default_model); the laya-auto release-check row; the work order; and four owner decisions (the small NPU row is a FastFlowLM checkpoint, not ours; catalogue vs table repos; auto as a reserved name vs a collection; whether the first code request may pull 18.6 GB).

Docs only — no code, no policy schema, no behaviour change. tools/copyright.py --check passes; the new file carries the Apache notice.

Refs: engine #186/#187 (routing RFC), #206 (GGUF scorer), #208 (HRX scorer), docs/laya.md, docs/hybrid.md.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit 45a5797)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Release Check The document describes a release-check row laya-auto in section 7, but does not include the actual test case or assertion logic that would validate the behavior described. This is a critical part of the design that needs to be implemented and tested to ensure correctness. ## 7. Gate: a release-check row `laya-auto` On a cold lemond (empty cache), three prompts — one code, one 2 KB document, one one-liner — each must land on the expected class, model and device, with `x-lemonade-route` naming the class rule, decision time under 100 ms, and the pull+load counted separately (so a regression in either is visible). A second immediate request must show no pull and no re-decision, and an explicit model name must produce no routing header at all. That is four assertions per prompt, which is the same shape as the existing `chat-hrx`/`chat-npu` rows. When `on_busy` ships (§4), the row gains a fifth case — a ticket held by another label makes the document prompt land on the NPU row, the header says `why: gpu_busy`, and the same prompt with the queue empty lands on `hrx` — because a fallback that only ever runs on one side of the condition is the classic untested branch. Unclear Owner Decision Section 9 lists several owner decisions that are left unresolved, including whether to use first-party NPU containers or FastFlowLM checkpoints, how to handle catalogue vs table naming, and how to manage the code class's first pull size. These decisions need to be made and documented to ensure consistent implementation. ## 9. Owner decisions 1. **The small NPU row is not ours.** Our only first-party NPU container on the Hub is `1bit-MONSTER/Qwen3.6-35B-A3B-NPU` (22.9 GB); the small NPU rows in `server_models.json` are FastFlowLM checkpoints on the pinned FLM v1.0.7. Is an FLM checkpoint "our model" for `chat-small` on the NPU, or do we publish a small Q4NX container of our own? 2. **Catalogue vs table.** The catalogue points at `unsloth/…` for Qwen3-Coder-30B-A3B and Qwen3.8-27B while we host first-party mirrors under `1bit-MONSTER`. The table names ours; confirm the row ids move with it. 3. **`auto`.** A reserved model name (needs a resolution hook in `Router`), or a shipped `collection.router` collection literally named `auto` (no code change, but it collides with any user model of that name)? 4. **The code class's first pull is 18.6 GB.** Start `code` on ZAYA1-8B (5.6 GB) and treat Qwen3-Coder as an escalation, or accept the big pull on the first code request?

[github-actions]

Persistent review updated to latest commit 8421ada

[bong-water-water-bong]

VERDICT: READY at 8421adaed. I ran the doc's "what exists today" section against the binary it names, and checked
every Hub claim; all of it holds.

The three probes, reproduced (~/wt/lemonade-46/fork/build/lemond, --version = 2026.46.0~1778.8888dd42,
exactly the build the doc cites; started on 127.0.0.1:13499 with an empty cache and config, killed afterwards — the
owner's :8088 embed server and the system lemond were not touched):

POST /v1/routing/validate, deterministic policy (keywords_any -> ZAYA1-8B, min_total_chars 2000 -> 27B, default 0.6B)
  "def f(x): return x"                200  0.6 ms  decision.route_to=1bit.ZAYA1-8B  matched_rule="code"  default_used=false
  2170-char document                  200  0.6 ms  decision.route_to=1bit.Qwen3.8-27B matched_rule="long" default_used=false
  one-line question                   200  0.6 ms  decision.route_to=1bit.Qwen3-0.6B  matched_rule=""      default_used=true
  (trace shows each condition: keywords_any true / false, then min_total_chars true / false)
POST /v1/chat/completions {"model":"auto"}   404  {"error":{"code":"model_not_found", …}}
POST /v1/routing/validate, {"type":"laya"}   400  {"error":"Invalid routing policy: unknown classifier type: laya"}

So the three claims that matter for the design are exactly right: the validator and decision format work today, auto
does not exist, and laya is not a classifier type. My timings are 0.6 ms rather than the doc's 9/4/3 ms — same
order, faster box; the argument ("25 ms, not a model load") is unaffected.

The Hub table, checked against the API. HF has 68 models under 1bit-MONSTER (the doc's count), and every
row's repo exists with the file it names:

1bit-MONSTER/Qwen3-0.6B-GGUF                      Qwen3-0.6B-Q8_0.gguf                            OK
1bit-MONSTER/ZAYA1-8B-GGUF                        zaya1-8b-Q4_K_M.gguf                            OK
1bit-MONSTER/Qwen3-Coder-30B-A3B-Instruct-GGUF    Qwen3-Coder-30B-A3B-Instruct-Q4_K_M.gguf        OK
1bit-MONSTER/Qwen3.8-27B-UD-Q4_K_XL-MTP           Qwen3.8-27B-UD-Q4_K_XL.gguf  + MTP/mtp-…Q4_0.gguf  OK / OK
1bit-MONSTER/Qwen3.6-35B-A3B-NPU                  model.q4nx                                      OK (7 files, dir-mode)

The cited symbols exist where the doc says (offsets of ±1 line move with the fork's commits, which is expected):
Router::resolve_model_name router.cpp:161, try_get_model_info :165, Router::load_model :850,
parse_classifier_configs with the model-type validation block at routing_policy_parser.cpp:417-440,
OnebitServer::load onebit_server.cpp:220 and the serve -m … --device … --alias … argv at :247-251,
struct ClassifierServices routing_policy.h:162.

Docs only, no policy schema or behaviour change; tools/copyright.py --check rc 0 and the new file carries the Apache
notice; docs/laya.md only gains a pointer to it.

Two notes, neither a gate:

1. The policy example's scorer.hf_repo is "mys/laya-typed-decisions-GGUF" — a placeholder for a scorer that is not
   published yet. Worth saying so in one word, since every other repo in that block is real and checked.
2. The doc says the classifier type is "unknown"; once this work exists, an outdated fork that rejects laya will
   look identical to a current one that rejects a typo. The version string in the error, or a GET /v1/routing/types,
   would separate "not implemented" from "misspelled" — a note for the implementation PR, not this one.

[bong-water-water-bong]

New commit d1fa3d5 (docs only, +46/−1): the routing table now states which device, not only which model, and makes the NPU a contention fallback rather than a speed choice.

Input (measured on this box by the Bonsai hybrid lane, ~/briefs/TBONSAI-HYBRID.md §0/§2, llama-bench, same file, -r 2): Ternary-Bonsai-2-27B, dense 27B-class — HRX0 124-152 tok/s prefill / 15.3 decode, against 13.9 / 3.05 on our own NPU route (bonsai2-27b-t2s2, 916e90d). HRX wins both halves (10.7-13.2× and 5.0×), and the post-reboot re-run with four lanes live (138.8 / 151.8 / 131.1 prefill, 13.9 decode) does not move it. So the "HRX prefill + NPU decode" split in docs/hybrid.md is not worth building for this model, and long_doc defaults to hrx.

What the note now specifies — an optional per-class on_busy {device, model, queue_depth, tctl_c, wait_s}:

• the signals are the ones the box already has: live tickets in the fleet fair GPU queue (the FIFO gpu-hold arbitrates) or a foreign token holder, and Tctl from k10temp temp1_input — the file gpu-hold's own start gate and thermal guard read;
• decided once at load time, never per request, and the decision header must carry device: npu, why: gpu_busy|gpu_hot, because the fallback costs 5× on decode and silence there would be a slowdown presented as normal;
• off for interactive classes — short/code would rather wait in the queue than answer 5× slower;
• thresholds are H0's to fill, not this note's: what a 512-token HRX decode costs in power/heat and what pinning the iGPU for 33 s costs the other lanes. Until that lands on_busy is absent from the shipped table, and §7 gains the busy case (queue held → NPU + why, queue empty → HRX) as its fifth assertion when it ships.

No code, no shipped table, no threshold invented in this commit.

[github-actions]

Persistent review updated to latest commit d1fa3d5

[bong-water-water-bong]

VERDICT: READY at d1fa3d5d6 (re-verdict: the two commits after 8421ada are content, not a main merge — 1f8852d + 8421ada are the doc, d1fa3d5 adds the "HRX by default, NPU only when the GPU is busy or hot" section, +46 doc lines).

Read: the measurement table (HRX0 124-152 prefill / 15.3 decode vs the NPU route 13.9 / 3.05 on the same file, -r 2, and the reboot re-run 138.8/151.8/131.1 with four lanes loading) is the kind of arithmetic that kills a design rather than promoting one, and the conclusion follows: device: hrx for the heavy class, the NPU as a contention fallback, not a speed one. The three properties are the ones that matter (decided once at load and said in the decision header, off for the interactive classes, thresholds left to H0 rather than guessed), and on_busy absent from the shipped table until H0 lands is the honest state.

One nit worth one sentence before merge: grep finds no code anywhere in the tree that reads laya-first.json or --laya-first — the whole format is still design (§Engine at the file's end lists it as work). So the new JSON example is an example of a file nothing parses yet, on_busy included. The section does say "absent from the shipped table", but a reader who lands on the field table first will not know that; a single line above the table ("nothing reads this file yet; the format below is the one the router will take") costs nothing and prevents the same trap #967 had to fix later.

[github-actions]

Persistent review updated to latest commit e0589e7

[github-actions]

Persistent review updated to latest commit 45a5797