[bong-water-water-bong]

Implements accepted RFC https://github.com/orgs/1bit-MONSTER/discussions/186.

Asked for a device, Laya picked ZINC for every request (it knows nothing about the hardware). Now it answers a question about the request, and a measured policy picks the device.

What changes

• Classifier (laya/route.h): long_doc by size (≥ 1,024 chars, no model call), else a three-way Laya choice among code / prose / short. typed-decisions/ checkpoint: 95.5% on 200 hand-labelled requests (code 100, prose 82, short 100, long_doc 100); ctest laya_classify fails below 90%.
• Policy (config/route-policy.json, compiled in; --route-policy FILE): class → device with the measured reason per row; below confidence 0.16 → default. Every class → Vulkan today: no class has a measured faster device, and llama-server ignores per-request speculative.* settings (verified on our pin and upstream master), so per-class drafters would need a second server holding a second copy of the model.
• serve --laya (checkpoint from $ONEBIT_LAYA_MODEL or ~/.local/share/1bit/laya) or --laya-model DIR: one decision per conversation (cached), X-1bit-Route: <class> <confidence> <device> on replies, each decision logged. Opt-in: --device auto without --laya is unchanged.
• 1bit route --classify; scripts/fetch-laya.sh defaults to the path --laya reads.

Measured (Strix Halo, Qwen3.8-27B Q4_0 + DFlash2, 1bit serve --device auto)

	1-token reply, first turn	later turn	decode code / prose / short
without --laya	498-518 ms	713-849 ms	50.5-50.7 / 26.7-26.9 / 19.0-19.2
--laya	1,008-1,048 ms	703-767 ms	50.6 / 26.7 / 18.9-19.1

Checks (strixhalo)

ctest 10/10 with -DONEBIT_LAYA_ROUTE_MODEL: smoke_serve, think_split, gguf_meta, laya_route_e2e (11/11: three classes to three fake devices, header, reuse on a later turn, missing-checkpoint error, npu-only), laya_classify (95.5% ≥ 90%), registry_gaps, registry_pins, npu_*.

🤖 Generated with Claude Code

[context7]

Docs7 for 1bit-monster/engine

Result	Status	Action
Deployment	➖ Not used	—
Content review	➖ Did not run. This site has no agent runs available this month. Wait for the monthly reset or check your Docs7 plan.	—

Commit dcec90d

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit dcec90d)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Race Condition in Class Cache The class cache (class_cache and class_order) in serve_child is protected by class_mu mutex, but there's a potential race condition during cache insertion. Specifically, when a new conversation key is added to the cache, the code first checks if the cache size is at the limit (class_cache.size() >= 4096) and removes the oldest entry if needed. However, this check and removal are not atomic with the insertion operation. If two threads attempt to insert simultaneously when the cache is at capacity, both might pass the size check and proceed to remove the oldest entry, potentially leading to one of the new entries being incorrectly evicted or causing a data race during the insertion. if (fresh) { std::lock_guard<std::mutex> lock(class_mu); if (class_cache.size() >= 4096) { // oldest conversation first class_cache.erase(class_order.front()); class_order.pop_front(); } if (class_cache.emplace(key, cls).second) class_order.push_back(key); } Incomplete Error Handling in Long Document Classification The classify_request function handles long documents by directly assigning the class "long_doc" without checking if the scorer is properly initialized or if there are any internal errors. If the scorer fails to initialize or encounters an error during its operation, this could lead to incorrect classification or silent failures, especially when the scorer is expected to be valid for other parts of the system. if (state.size() >= kLongDocChars) { out.label = "long_doc"; out.confidence = 1.0f; out.probabilities = {{"long_doc", 1.0f}}; return out; } Potential Invalid Device Selection The RoutePolicy::pick function may return an empty string if the fallback device is not in the candidate list and the candidates list is empty. This could happen if the policy is misconfigured or if the device list is empty, leading to an invalid routing decision. While the code checks for fallback device availability, it doesn't handle the case where the fallback device is not in the candidates list and the candidates list is empty, which could lead to a crash or undefined behavior. std::string RoutePolicy::pick(const std::string& cls, float confidence, const std::vector<std::string>& candidates) const { auto have = [&](const std::string& d) { return std::find(candidates.begin(), candidates.end(), d) != candidates.end(); }; if (confidence >= min_confidence) if (const auto it = device.find(cls); it != device.end() && have(it->second)) return it->second; if (have(fallback)) return fallback; return candidates.empty() ? std::string() : candidates.front(); }

---

⚠️ Review coverage: The following files were not included in this review because of the token budget:

• tests/laya_route_cases.json

[github-actions]

Persistent review updated to latest commit dcec90d