npu plan: the KV cache contract (P7a), and the arithmetic it rests on gated in CI

CMakeLists.txt

@@ -856,6 +856,12 @@ add_dependencies(npu_sampling_test npu_rope_llama3_test)
 add_executable(hadamard_rotate_test tests/hadamard_rotate_test.cpp)
 target_include_directories(hadamard_rotate_test PRIVATE npu/forward/include)
 add_test(NAME hadamard_rotate COMMAND hadamard_rotate_test)
+# The arithmetic P7 (#806) part A rests on: (QH)(KH)^T = QK^T to f32 rounding, and what the rotation is
+# worth against an f16 KV cache (attention-weight KLD) with and without outlier channels.  Header only, no
+# device, no model: runs in CI.
+add_executable(npu_kv_int8_test tests/npu_kv_int8_test.cpp)
+target_include_directories(npu_kv_int8_test PRIVATE npu/forward/include)
+add_test(NAME npu_kv_int8 COMMAND npu_kv_int8_test)
 # The A-row quantiser every NPU GEMM runs on the host before its launch (npu/forward/include/quant_i8.h): its
 # AVX2 path must be byte-identical to the shipped scalar loop, including the .5 boundaries, the NaN/inf inputs
 # and the values past 2^31 that the (int) cast saturates.  Header only, no device, no model: runs in CI.
@@ -873,7 +879,7 @@ add_test(NAME attn_math COMMAND attn_math_test)
 # ci.yml's build step names its targets explicitly and needs the `workflow` scope to extend, so bind the new
 # binary to a target the list already builds (same reason as npu_moe_test above); drop this line when the
 # workflow is next touched with that scope and add hadamard_rotate_test to the list itself.
-add_dependencies(npu_forward_loader_test hadamard_rotate_test quant_i8_row_test attn_math_test)
+add_dependencies(npu_forward_loader_test hadamard_rotate_test quant_i8_row_test attn_math_test npu_kv_int8_test)
 # The window11 gate's comparison policy (engine#834): no tolerance without a measurement behind it, the
 # statistic named, and a control pair setting the tolerance from its own spread.  Shell + the fixtures it
 # writes itself, so there is no target for the build list to miss.

docs/npu-plan.md

@@ -61,6 +61,8 @@ tools/npu_plan.py <model dir> [--kernels DIR] [--out PATH] [--check] [--print]
 | `decode.sealed_elf` | pb16i only: the sealed ELF the decode roles run |
 | `kernels.tag` | artifact-name suffix: `""`, `_b8`, `_pb16i`, `_t2s` or `_t2s2`; must agree with the container layout and `container.pb16i` |
 | `attention.mode` | `causal` (default) or `noncausal_full_attention`. The latter is a prompt-encoding mode: every full-attention layer sees the whole prompt, and the forward refuses to generate with it |
+| `kv.key`, `kv.value` | what the KV cache is stored as, per side: `f16` (the meaning of an absent `kv` block, and of every plan written before P7) or `int8`. `int8` is int8 with **one fp32 scale per (head, token)**, taken *after* a per-head Hadamard rotation — the rotation is what makes one scale affordable, because it collapses the per-channel magnitude spread the scale would otherwise have to cover. `key` is the side that costs no un-rotation (`(QH)(KH)ᵀ = QKᵀ`), so storing K rotated and rotating Q needs no inverse transform; `value` is a separate choice because an int8 V needs one on the output, which is why K-only is the cheap variant to prove the path with. This is a description: a target without the capability must refuse `int8` **by name**, never serve f16 in its place |
+| `kv.hadamard.block`, `kv.hadamard.signs` | the rotation itself: `block` is `0` (no rotation) or a power of two in `[2, 65536]` — the head width the rotation runs over — and `signs` is an optional **bare file name** (no directory part, under the model dir) for per-channel signs. A `signs` file with `block: 0` is refused rather than ignored |
 | `moe.batched_expert_gu` | `{enabled, K, N}`; when enabled, K and N must be named |
 | `layers[]` | one entry per layer, in order: `index`, `role` (`attn`/`gdn`/`moe`/`mla`), and projections `qkv {K, N, fused}`, `attn_gate {K, N}`, `gate_up {K, N}`, `down {K, N}`. The count must equal `model.layers`. The projection shapes are recorded for P4’s `NpuModel`; the current forward reads only the count (for its [plan] log line) and derives designs from the container |
 
@@ -81,6 +83,9 @@ The loader never substitutes a default for something it cannot honour; every ref
   `_t2s`/`_t2s2` require `t2s`);
 - `decode.dirs` with an empty array, a `root` other than `model`/`kernels`, or a path that is empty,
   absolute, contains `..` or a backslash;
+- `kv.key`/`kv.value` outside `f16`/`int8`; a `kv.hadamard.block` that is not `0` or a power of two in
+  `[2, 65536]`; a `kv.hadamard.signs` that is a path rather than a bare name, or that names a file while
+  `kv.hadamard.block` is `0`;
 - `moe.batched_expert_gu.enabled` without positive K and N;
 - `layers` missing/out of order, a role outside the four, a projection naming only one of K and N, or a
   layer count that disagrees with `model.layers`;
@@ -90,6 +95,30 @@ The loader never substitutes a default for something it cannot honour; every ref
 The forward adds one more check on load: `kernels.tag` must equal the tag the container itself implies
 (`q4nx_forward.cpp:510-539`), so a plan and a container cannot disagree about the artifact family.
 
+## What the int8 KV contract is worth, measured
+
+The two claims `kv` rests on are arithmetic, not device behaviour, so they are gated in CI with no model and
+no device (`ctest -R npu_kv_int8`, `tests/npu_kv_int8_test.cpp`). Attention-weight **KLD against the f16
+cache**, head width 256, one scale per (head, token), 512 keys, 8 draws per cell:
+
+| KV vectors | f16 + rotation | int8 K, unrotated | int8 K + rotation | rotation's worth |
+|---|---:|---:|---:|---:|
+| uniform (control) | 3.9e-15 | 2.5e-05 | 2.4e-05 | **1.0x** — nothing to concentrate, nothing to claim |
+| 8x outlier channels | 8.6e-15 | 4.5e-04 | 4.7e-05 | **9.6x** |
+| 30x outlier channels | 4.5e-14 | **8.2e-03** | **1.5e-04** | **54.5x** |
+
+Two readings matter. The rotation is **free**: rotating both sides and taking the dot product preserves the
+score to 1.6e-09 relative, which is *smaller* than one f32 dot product's own rounding error (0.01x of it), so
+a rotated f16 cache is indistinguishable from today's. And the rotation is what **keeps int8 inside the
+plan's gate**: at 30x outliers an unrotated int8 K cache lands at 8.2e-03, an order of magnitude outside
+1e-03, while the rotated one stays at 1.5e-04. At 8x outliers the unrotated cache is already inside the gate
+— so on a model whose channels are mild, the rotation buys accuracy this gate does not ask for, and on one
+whose channels are not, it is the difference between passing and failing.
+
+What this does **not** establish is the gate itself: the vectors are synthetic and generated to the outlier
+structure this page describes, and #806's KLD is taken on real KV from a real model on the device. The test
+bounds the mechanism; the model run decides.
+
 ## How the forward consumes it
 
 At init the forward looks for `<model dir>/plan.json`. If present it loads it and uses the plan for the
@@ -141,4 +170,10 @@ layers[0]     {"index": 0, "role": "gdn", "qkv": {"K": 5120, "N": 10240, "fused"
   until that interface lands.
 - **P5** (#804): Hexagon target (t2 gemv on HVX/HMX, QEMU v73 oracle) has no engine rows yet.
 - **P6** (#805): the shared drafter protocol is on main; tau=0.6 and Hexagon acceptance rows are still missing.
-- **P7** (#806): int8 KV with per-head Hadamard and NPU prefill chunks; the plan has no `kv.dtype` field yet.
+- **P7** (#806): the `kv` contract above is in the plan and its arithmetic is gated in CI (`ctest -R npu_kv_int8`,
+  which needs no device: see below). Still open, in the order it can be taken: the cache **writer/reader** in the
+  forward (an int8 K store plus its scale sidecar, the prefix-cache snapshot's byte accounting with it), the **KLD
+  gate on real KV from a model on the device** — the CI test bounds the mechanism on synthetic vectors, not the
+  model — and **part B**, prefill in 1024/2048-token chunks with block-fp expansion, which waits on P3's on-core
+  expansion, P4's `NpuModel`, and the `M>16` decision (prefill *is* M≫16; nothing in P7b asks for M>16 in the
+  decode path).

npu/plan/plan.h

@@ -138,6 +138,21 @@ struct Plan {
     std::string tile_order = "engine-1";
     std::string attention_mode = "causal";
     bool noncausal() const { return attention_mode == "noncausal_full_attention"; }
+    // P7 (#806) part A: what the KV cache is stored as.  `f16` is every plan written before this field
+    // existed, and stays the default for that reason.  `int8` stores that side as int8 with one fp32
+    // scale per (head, token), taken AFTER a per-head Hadamard rotation -- the rotation is what makes one
+    // scale affordable, because it collapses the per-channel magnitude spread the quantizer would
+    // otherwise have to cover (docs/npu-plan.md).  `key` is the side that pays no un-rotation, since
+    // (QH)(KH)^T = QK^T; `value` is a separate choice because an int8 V needs its own inverse transform
+    // on the output, so K-only is the cheapest variant to prove the path with.
+    //
+    // This is a description, not an implementation: a target that cannot honour `int8` must refuse it by
+    // name.  Falling back to f16 would answer with the right text and a different cache contract, which
+    // is the failure mode the plan layer exists to remove.
+    std::string kv_key = "f16", kv_value = "f16";
+    int kv_hadamard_block = 0;       // 0 = no rotation
+    std::string kv_hadamard_signs;   // optional per-channel sign file, bare name under the model dir
+    bool kv_int8() const { return kv_key == "int8" || kv_value == "int8"; }
     // Where the decode family is looked for, in order: which given path it hangs off, and the
     // relative directory under it ("." = that directory itself).
     struct DecodeDir { bool under_model = false; std::string path; };
@@ -211,6 +226,40 @@ struct Plan {
                 throw PlanError("plan: " + path + " attention.mode \"" + p.attention_mode +
                                 "\" is not causal or noncausal_full_attention");
         }
+        // Optional and defaulted on purpose, for the same reason as attention.mode: a plan written
+        // before P7 existed describes f16 KV, which is what every plan meant then.
+        if (j.contains("kv")) {
+            const auto& kv = obj(j, "kv", path);
+            // Type before value: a number where a dtype belongs is a refusal that names the field, not a
+            // JSON exception escaping the loader.
+            for (const char* side : {"key", "value"})
+                if (kv.contains(side) && !kv.at(side).is_string())
+                    throw PlanError("plan: " + path + " kv." + side + " is not a string");
+            p.kv_key = kv.value("key", "f16");
+            p.kv_value = kv.value("value", "f16");
+            if (p.kv_key != "f16" && p.kv_key != "int8")
+                throw PlanError("plan: " + path + " kv.key \"" + p.kv_key + "\" is not f16 or int8");
+            if (p.kv_value != "f16" && p.kv_value != "int8")
+                throw PlanError("plan: " + path + " kv.value \"" + p.kv_value + "\" is not f16 or int8");
+            if (kv.contains("hadamard")) {
+                const auto& h = obj(kv, "hadamard", path);
+                p.kv_hadamard_block = positive_or_zero(h, "block", path);
+                const int b = p.kv_hadamard_block;
+                if (b != 0 && (b < 2 || b > 65536 || (b & (b - 1)) != 0))
+                    throw PlanError("plan: " + path + " kv.hadamard.block " + std::to_string(b) +
+                                    " is not 0 or a power of two in [2, 65536]");
+                p.kv_hadamard_signs = h.value("signs", "");
+                // A bare file name, never a path: a plan must not be able to point the loader at a file
+                // the command line did not name (the same rule decode.dirs follows).
+                if (p.kv_hadamard_signs.find('/') != std::string::npos ||
+                    p.kv_hadamard_signs.find("..") != std::string::npos)
+                    throw PlanError("plan: " + path + " kv.hadamard.signs \"" + p.kv_hadamard_signs +
+                                    "\" must be a bare file name (no directory part)");
+                if (b == 0 && !p.kv_hadamard_signs.empty())
+                    throw PlanError("plan: " + path +
+                                    " kv.hadamard.signs names a file but kv.hadamard.block is 0 (no rotation)");
+            }
+        }
         if (j.contains("kernels")) {
             const auto& k = obj(j, "kernels", path);
             p.tag = k.value("tag", "");