Skip to content

npu: a rope_scaling type the route does not implement is served plain RoPE (yarn/linear/longrope), silently #1087

Closed

Description

@bong-water-water-bong

Two front doors accept a model that declares a RoPE scaling block they do not apply, and neither says so:

  1. tools/gguf_to_q4nx.py (the GGUF → packed-container route, 1bit npu-run <gguf>) reads *.rope.freq_base and
    nothing else, so rope.scaling.type = yarn | linear | longrope | dynamic is dropped. The container is written with
    rope_theta alone and the lane computes theta^(-i/half) (npu/pack.cpp fill_rope), which is right only inside the
    model's original window.
  2. npu/forward implements two rules — llama3 (engine#1048) and LongRoPE short_factor — and reads nothing else.
    A container whose config.json declares yarn (or any other type) loads and serves as if the block were absent.

Why it matters: the failure is silent and looks like a model that is merely a bit dumb past the original window. The
size of it is measured for the case that is implemented — npu/forward/include/q4nx_rope.h: ignoring the llama3 rule
on Llama-3.2-1B gives KL 0.0058 at ctx 104, 0.0107 at 754, 0.0252 at 1497 against the same checkpoint with it applied.
A YaRN model served plain has the same shape of error and no line in any log to say so.

Refusing is the honest behaviour, not implementing every rule: YaRN's attention-factor schedule, dynamic NTK and
look-up scaling are each a feature of their own, and guessing one would be worse than a named refusal.

Found while closing the #844 comment item; not the same defect as #1048 (that one had an implementation and no wiring,
these have no implementation at all).

Cc the pack/convert path (OGALOAD lane).

Activity

  1. added a commit that references this issue on Nov 5, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions