[bong-water-water-bong]

A rope scaling the route does not apply is now refused by name, at both front doors

Closes #1087.

Two entry points accepted a model that declares a RoPE scaling block they do not implement, and neither said so:

• tools/gguf_to_q4nx.py (the GGUF → packed-container route) read *.rope.freq_base only, so
  rope.scaling.type = yarn | linear | longrope | dynamic was dropped: the container was written with rope_theta
  alone and the packed lane computes theta^(-i/half) (npu/pack.cpp fill_rope), which is right only inside the
  model's original window.
• npu/forward implements llama3 (npu: a pack that declares rope_type "llama3" is served plain RoPE — 17 of 32 dim pairs run up to 32x fast (Llama-3.x) #1048) and LongRoPE short_factor, and nothing read any other type, so a
  container whose config.json declares one was loaded and served as if the block were absent.

The failure mode is the reason this is worth a PR at all: the reply stays fluent and is wrong past the original
window, and no line in any log says the model was served something other than it asked for. The size of it is measured
for the case that is implemented — npu/forward/include/q4nx_rope.h: ignoring the llama3 rule on Llama-3.2-1B gives
KL 0.0058 at ctx 104, 0.0107 at 754, 0.0252 at 1497 against the same checkpoint with it applied.

Refuse rather than implement: YaRN's attention-factor schedule, dynamic NTK and look-up scaling are each a feature of
their own, and a guessed rule is worse than a named refusal.

The change

• npu/forward/include/q4nx_rope.h — two header-only functions next to the llama3 rule:
  q4nx_rope_declared_type() (the declared type from config.json, both HF spellings rope_type/type, "" when the
  block is absent or null) and q4nx_rope_type_supported() (""/default/none = plain; llama3 only with usable
  numbers; longrope only with the short_factor the forward reads; everything else refused).
• npu/forward/src/q4nx_forward.cpp — init_impl calls them after the two existing readers and fails the load with a
  message naming the type. That is before any kernel or device use, and it is the path init_host shares, so the
  offline host-GEMM route refuses the same way.
• tools/gguf_to_q4nx.py — refuse_unsupported_rope(), called at the top of pack() before the output directory is
  created (so no half-written container) and again in config() (so no other command can bypass it).
• Tests: tests/npu_rope_llama3_test.c gains a 13-case table over the config shapes that exist in the wild, including
  the trap this reader could have fallen into — "rope_scaling": null followed by another block that does carry a
  "type" must still declare nothing. tests/gguf_to_q4nx_test.py packs synthetic GGUF writer output for all three
  refused types and asserts a non-zero exit, the type named in the message, and no model.q4nx left behind.

Measured before / after, same reproducer

Converter, tests/gguf_to_q4nx_test.py's synthetic qwen3 GGUF plus w.add_rope_scaling_type(YARN):

OLD  rc=0   wrote /tmp/rope-repro/out-old/model.q4nx (716296 bytes)
            config.json rope fields: {'rope_theta': 1000000.0, 'rope_scaling': None}   <- the block is gone
NEW  rc=1   qwen3.rope.scaling.type is 'yarn': this converter has no RoPE scaling rule, so the container would be
            served plain RoPE and reply wrongly past the model's original window. ... (engine#1087)
            no output directory created

Forward, host-GEMM route (NPU_INFER_HOST_GEMM=1, no device opened) against a copy of a real container whose
config.json says {"type": "yarn", "factor": 4.0, ...} instead of the llama3 block:

OLD no guard: [semantics] per-type rope: full(theta=500000 rot=0) sliding(theta=0 rot=0) rope_type_named=0
1bit forward-serve: yarn-ctr ready on http://127.0.0.1:18099 (full ELF) <- served, block ignored
NEW guard: 1bit forward-serve: forward init failed: config.json declares rope scaling "yarn", which this
forward does not implement (only llama3 and longrope short_factor are), so it would be served plain
RoPE and reply wrongly past the model's original window (engine#1048, #1087) rc 1
CONTROL the same container with its real llama3 block: "ctr-llama ready on http://127.0.0.1:18098"

Both runs used `NPU_INFER_HOST_GEMM=1`, so `init_host` was the path and the NPU was never opened (this box's array was
held by another lane throughout; the engine's `[npu] exclusive device lock` was taken and released inside each run).

Tests:

$ ./build/npu_rope_llama3_test
all llama3 rope checks passed
$ python3 tests/gguf_to_q4nx_test.py
ok   packing a yarn-scaled GGUF exits non-zero and names the type (rc 1)
ok   no model.q4nx was written for it
ok   packing a linear-scaled GGUF exits non-zero and names the type (rc 1)
ok   no model.q4nx was written for it
ok   packing a longrope-scaled GGUF exits non-zero and names the type (rc 1)
ok   no model.q4nx was written for it
PASS
$ ctest -R "npu_rope_llama3|npu_forward_loader"
100% tests passed, 0 tests failed out of 2
$ python3 tools/copyright.py --check
rc 0

Not a regression for anything on this box

Every GGUF here declares no scaling block (checked with the gguf reader, rope.scaling.type None for all 12:
Qwen3-0.6B/4B, Qwen3-Coder-30B-A3B, Qwen3.5-4B, Qwen3.6-35B-A3B, bonsai2, zaya1-8b, BlackMamba-2.8B, Zamba2-2.7B,
ZR1-1.5B), and the two pack-imported families (rope_type: llama3 and no block at all) are on the supported side of
the decision, which the table test pins.

Scope, stated rather than implied

npu/pack.cpp fill_rope (the packed lane route) still has no scaling rule of its own; this PR only stops a GGUF with a
declared scaling from reaching it. Implementing a rule there is the other half of #1087 and is not started here — worth
noting that the lane's fill_rope is hard-wired to 128 dim pairs (64 cos + 64 sin), so it could only ever serve a
head_dim-128 model anyway.

Nothing merged, tagged or published.

[bong-water-water-bong]

VERDICT: READY at a1c4cac — behaviour verified at 0992f65 (the only change on top is docs/npu.md +11/−0), both doors reproduce, and the guard does not cost a good model its load.

Converter door, reproduced with my own GGUF writer rather than the PR's fixture — six tiny GGUFs, qwen3.rope.scaling.type set to each value, converted at this head:

plain     rc=1  out=no-claim-holds   (gate passes; fails later: my synthetic file has no tensors)
none      rc=1  out=yes              (gate passes — 'none' is plain rope)
default   rc=1  out=yes              (gate passes)
yarn      rc=1  out=no   qwen3.rope.scaling.type is 'yarn': this converter has no RoPE scaling rule ...
linear    rc=1  out=no   ... 'linear' ...
longrope  rc=1  out=no   ... 'longrope' ...
dynamic   rc=1  out=no   ... 'dynamic' ...

Every refused case names the type and leaves no output directory — the ordering claim (refuse_unsupported_rope() before os.makedirs) holds in the code and in the run, and config() calls it too, so verify cannot bypass it.

Forward door, on real containers, under one npu-hold ticket (NPU_INFER_HOST_GEMM=1, no device dispatch):

yarn block        rc=1  config.json declares rope scaling "yarn", which this forward does not implement
                        (only llama3 and longrope short_factor are), so it would be served plain RoPE and
                        reply wrongly past the model's original window (engine#1048, #1087)
llama3, factor 0  rc=1  declares rope_type "llama3" but its numbers are not usable (factor > 0, ...)
CONTROL valid llama3 block   ->  ctrLlama ready on http://127.0.0.1:18612 (full ELF)      (served, then stopped)
CONTROL no block             ->  844-out-clean ready on http://127.0.0.1:18613 (full ELF) (served, then stopped)

The two controls are the half that matters for a refusal PR: a legitimate llama3 container and a plain one still load and serve, so this is not a guard that refuses the models it claims to support. The unusable-llama3 case gets its own message instead of the generic one, which is the right split — it is a broken block, not an unimplemented rule.

./npu_rope_llama3_test → all llama3 rope checks passed; ctest -R "npu_rope_llama3|npu_forward_loader" → 2/2; build clean at -DONEBIT_NPU=ON, 17 warnings, none from q4nx_rope.h, q4nx_forward.cpp or the new test.

Two nits, both latent (neither is reachable with any file on this box) — the guard is a substring scanner and both edges are in it:

1. It can fail open. q4nx_rope_declared_type() searches 800 bytes past the "rope_scaling" key; a rope_type that sits beyond that window reads as no block, i.e. supported=1, i.e. served plain — the exact silent failure the PR exists to stop. Probed directly:

E  rope_type 900B into the block   {"rope_scaling": {"pad": "xxx…900", "rope_type": "yarn"}}   declared=(none)  supported=1
G  rope_type 700B into the block   same with 700 bytes                                          declared=yarn    supported=0

Reachable in principle for a LongRoPE/YaRN block serialised with its factor arrays before the type key; not reachable here — the only config on this box with a block is Llama-3.2-1B and its rope_type is at +145 bytes, and the ordering (factor, high_freq_factor, low_freq_factor, original_max_position_embeddings, then rope_type) is worth knowing because it is the shape that would eventually cross 800.
2. It can false-positive. The "type" fallback is searched from the key, not inside the object, so an object with no type followed within 800 bytes by another block's "type" declares that neighbour's type: {"rope_scaling": {"factor": 8.0}, "quantization_config": {"type": "fp8"}} → declared=fp8, refused. Fails closed, but with a confusing name in the message.

Both are fixed by bounding the search to the object's matching } instead of a fixed 800 bytes — a dozen lines, and it makes the scanner's window a property of the JSON rather than a magic number. Not a blocker: the null trap that is reachable (every config without a block) is handled correctly, and {"rope_scaling": null, "other": {"type": "yarn"}} reads as no block in my probe too.

Scope the PR states and I agree with: npu/pack.cpp fill_rope (the packed lane) still has no scaling rule — this only stops a scaled GGUF reaching it. Worth saying plainly in the PR body that the packed lane would also need one, which it does.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1087 - Partially compliant Compliant requirements: The GGUF conversion tool now refuses unsupported rope scaling types with clear error messages The forward route now checks for supported rope scaling types and refuses unsupported ones Clear error messages are provided indicating which scaling type is not supported Documentation has been updated to reflect this behavior Non-compliant requirements: None Requires further human verification: None 1048 - Partially compliant Compliant requirements: The forward route now properly implements llama3 rope scaling The forward route now properly implements LongRoPE short_factor scaling The forward route now reads rope_scaling parameters from config.json The forward route now builds the rope table with the correct three-branch rule for llama3 scaling Tests have been added to verify the llama3 scaling implementation Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Rope Scaling Refusal Logic The new rope scaling refusal logic in q4nx_forward.cpp introduces a new check that reads the entire config.json file to determine the declared rope scaling type. While this approach works, it could be inefficient as it reads the entire file content into memory. A more efficient approach would be to parse only the necessary JSON fields, but given the current implementation, this is a reasonable trade-off for correctness. { // A declared scaling this forward does not apply is refused BY NAME (engine#1087): serving it // would run plain rope, stay fluent and be wrong past the original window, with nothing in the // log to say so. "" (no rope_scaling block) is every model that never declared one. std::ifstream f(cfg_path); std::string t((std::istreambuf_iterator<char>(f)), std::istreambuf_iterator<char>()); char declared[64]; q4nx_rope_declared_type(t.c_str(), declared, (int)sizeof declared); if (!q4nx_rope_type_supported(declared, &rope_llama3_, rope_scale_.empty() ? 0 : 1)) { if (!std::strcmp(declared, "llama3")) err_ = "config.json declares rope_type \"llama3\" but its numbers are not usable (factor > 0, " "original_max_position_embeddings > 0, high_freq_factor > low_freq_factor), so the " "scaling cannot be applied; serving it would run plain RoPE"; else err_ = std::string("config.json declares rope scaling \"") + declared + "\", which this forward does not implement (only llama3 and longrope short_factor " "are), so it would be served plain RoPE and reply wrongly past the model's " "original window (engine#1048, #1087)"; return false; } } Early Refusal of Rope Scaling The refuse_unsupported_rope function in gguf_to_q4nx.py now prevents conversion of GGUF files with unsupported rope scaling types. This is a good change that prevents silent failures, but it's worth noting that this check happens before any output directory is created, which is the correct behavior to avoid half-written containers. def refuse_unsupported_rope(r): """A scaling this route cannot apply is refused before anything is written (engine#1087). A declared block that is dropped is worse than a failed conversion: the container would carry rope_theta and nothing else, be served plain RoPE, stay fluent and be wrong past the model's original window, with no line anywhere saying so. This route has no scaling rule at all (the packed lane computes theta^(-i/half) and nothing else), so every declared type is refused by name. The forward implements llama3 and longrope short_factor, but only for a container whose config.json carries the block, which is a different route (1bit npu-pack import, or a checkpoint's own config). """ arch = field(r, "general.architecture") scaling = field(r, f"{arch}.rope.scaling.type") if scaling is not None and str(scaling) not in ("none", "default", ""): raise SystemExit( f"{arch}.rope.scaling.type is {str(scaling)!r}: this converter has no RoPE scaling rule, so the " f"container would be served plain RoPE and reply wrongly past the model's original window. " f"Convert from the checkpoint with its config.json (the forward route reads rope_scaling), or " f"file the scaling type you need (engine#1087)") JSON Parsing for Rope Type Detection The q4nx_rope_declared_type function in q4nx_rope.h implements a custom JSON parser to extract the rope scaling type. While this works, it's a complex function that could be error-prone. However, since it's a header-only function and the tests cover various cases, it should be reasonably robust. static inline void q4nx_rope_declared_type(const char* json, char* out, int cap) { if (!out || cap <= 0) return; out[0] = '\0'; if (!json) return; const char* rs = strstr(json, "\"rope_scaling\""); if (!rs) return; { /* "rope_scaling": null declares nothing — and must not let the search below find some other * block's "type": that is the one way this reader could refuse a model that asked for nothing. */ const char* c = strchr(rs, ':'); if (!c) return; const char* v = c + 1; while (*v == ' ' || *v == '\t' || *v == '\n' || *v == '\r') ++v; if (*v != '{') return; } /* the block is a flat object in every config we read; 800 bytes is past its closing brace */ const char* end = rs + 800; const char* p = strstr(rs, "\"rope_type\""); /* current HF spelling */ if (!p || p > end) p = strstr(rs, "\"type\""); /* pre-4.45 spelling: {"type": "yarn"} */ if (!p || p > end) return; const char* c = strchr(p, ':'); if (!c || c > end) return; const char* q = strchr(c, '"'); if (!q || q > end) return; const char* e = strchr(q + 1, '"'); if (!e) return; int n = (int)(e - q - 1); if (n > cap - 1) n = cap - 1; memcpy(out, q + 1, (size_t)n); out[n] = '\0'; }

[bong-water-water-bong]

Both nits fixed at dac3a06 — the fail-open one first, because a guard that reads "no block" for a block is the
failure the PR exists to stop.

q4nx_rope_declared_type() now bounds its search by the block's own matching } instead of 800 bytes, matched
through strings (so a } inside a string, a nested array or a nested object cannot end the block early), and an
unterminated block declares nothing rather than guessing. The window is a property of the JSON now, not a magic number.

Your two probes, plus two more, are tests in tests/npu_rope_llama3_test.c. Three of them fail on the old scanner
(same test file, header checked out from a1c4cac, rebuilt):

OLD:  FAIL a type 1,000 bytes into the block is still read (fails closed)       <- your nit 1, fails OPEN
      FAIL ... and is refused
      FAIL a block with no type does not take a neighbour's type                <- your nit 2
      FAIL ... and is not refused
      FAIL an unterminated block declares nothing rather than guessing
NEW:  all llama3 rope checks passed

End to end at this head (NPU_INFER_HOST_GEMM=1, no device dispatch, each run stopped with SIGTERM):

yarn container                     rc=1  declares rope scaling "yarn" ... it would be served plain RoPE ...
yarn block, 1,000 B pad before type rc=1  same message        <- this one LOADED before this commit (fail-open)
CONTROL real llama3 container      ->    ctr-llama ready on http://127.0.0.1:18091 (full ELF)

The control still serves, so the tightened scanner did not turn into a guard that refuses good models. ctest -R npu_rope_llama3 2/2, copyright.py --check rc 0, and the converter half is untouched by this commit.

One process note for whoever reads the log: my first attempt at the padded-block run bound port 18097 for its 11 s
lifetime before I killed it. Nothing was listening there and the run was in host-GEMM mode, but that port is the
owner's and this lane must never bind it — the later runs used 18090/18091. Recorded in the lane state file too.

[bong-water-water-bong]

VERDICT: READY at dac3a06 — both nits are fixed the way I asked, the fix has teeth, and nothing that worked before changed.

Re-ran my own probe suite against the new scanner, old vs new, same nine inputs:

                                    OLD (0992f65)          NEW (dac3a06)
E rope_type 900B into the block     declared=(none) sup=1  declared=yarn   sup=0     <- the fail-open, closed
F no type, neighbour "type": fp8    declared=fp8    sup=0  declared=(none) sup=1     <- the false refusal, closed
A no block / B yarn / C null trap / D `{}` / G 700B / H text_config / I longrope      <- all unchanged

So it fixes exactly the two directions I raised and leaves the seven cases that were already right alone.

The regression guard has teeth, which is the part I actually care about. Compiling the new test file against the old header fails, and against the new one passes:

$ gcc -I<old header> tests/npu_rope_llama3_test.c -lm && ./a.out
FAIL a type 1,000 bytes into the block is still read (fails closed)
FAIL ... and is refused
FAIL a block with no type does not take a neighbour's type
FAIL ... and is not refused
FAIL an unterminated block declares nothing rather than guessing
rc=1
$ gcc -I<new header> … && ./a.out
all llama3 rope checks passed      rc=0

Three cases, five assertions, all of which the old scanner answered wrongly — so this is a test that would have caught the bug rather than a test that describes the fix.

Edges the new brace-walk gets right, probed directly because a hand-written JSON scanner lives or dies on them: a } inside a string value ("note": "a } here" → still yarn), an escaped quote before the block's brace (yarn → linear case correct), a nested object before the type key (dynamic), and a type value containing a brace (ya}rn). Braces are matched through strings with escape tracking, so the block's window is its own closing brace rather than a magic 800.

One residual direction, stated because it is the last one: an unterminated block ({"rope_scaling": {"rope_type": "yarn" and then end-of-file) declares nothing, i.e. fails open. I would not change it — a config that truncated mid-block is a broken file that the rest of the loader also reads, and the code's own comment ("nothing trustworthy to refuse") is a defensible call. Worth knowing, not worth a third round.

Everything from my READY at a1c4cac still stands unchanged: the converter refuses yarn/linear/longrope/dynamic by name with no output directory, none/default/absent pass, the forward refuses with a named message at load (host-GEMM, device never opened), and the two controls — a valid llama3 block and a plain container — still load and serve. The delta since then is the scanner, its tests and docs/npu.md; no behaviour outside the two edge cases moved.

[bong-water-water-bong]

A branch-hygiene lane merged main into this branch at 06:21 (528c4bf, merge of 37577e6), so the head is no longer
the dac3a06 that was reviewed. Re-verified at the merged head rather than assuming a clean textual merge is a clean
semantic one:

• the guard is intact — q4nx_rope_declared_type/q4nx_rope_type_supported are still called from init_impl (2 refs in
  q4nx_forward.cpp, the definitions in q4nx_rope.h), and nothing in the merge touched init_impl;
• main's change in the same file is step_batch's [bl] phase-counter ordering (npu: the verify batch's interior is measured per phase (new [bs] line), and the [lh] table stops freezing on its first call #1056 follow-up), a different function;
• onebit_npu_forward_syntax (the object CI compiles without the NPU runtime) builds clean at this head;
• ./build/npu_rope_llama3_test → all llama3 rope checks passed;
• python3 tests/gguf_to_q4nx_test.py → PASS, with all three scaling types still refused by name;
• the docs/npu.md paragraph is still in place (docs/npu.md:366).

build CI is running on 528c4bf; the last one on dac3a06 was green. Nothing else changed on the branch.