[bong-water-water-bong]

P7a groundwork: the KV cache contract in the plan, and the arithmetic it rests on measured in CI

Scoped first PR for #806 (plan P7: kv.dtype int8 + per-head Hadamard, chunked NPU prefill) — per the plan-layer rule in #799, a roadmap issue gets a scoped first step plus a comment on the issue listing what remains. This does not close #806.

Rule 1 of tonight's brief says c31 gates come first, and this PR is deliberately CPU-only: no GPU ticket, no NPU ticket, no device time at all.

Why this is the right first step

#806's own last comment names the gap: the design audit "lives in a private brief (~/briefs/DESIGN-P7-…md), not in this repository", and the plan layer can't describe an int8 KV cache because docs/npu-plan.md says in its own words "the plan has no kv.dtype field yet". P7's remaining work needs P4 (NpuModel/NpuTarget, PR #808) and the private npu-kernels repo, neither of which this lane can land tonight. What it does not need any of that for is the contract and the arithmetic — that is, the part that can be wrong silently.

What lands

• npu/plan/plan.h — kv {key, value, hadamard {block, signs}}, defaulting to f16/no rotation so every plan written before the field keeps loading and means exactly what it meant. int8 means one fp32 scale per (head, token) taken after a per-head Hadamard rotation. Validated with named refusals in the loader's existing style: an unknown dtype, a rotation block that is not 0 or a power of two in [2, 65536], a signs entry that is a path rather than a bare name, signs with no rotation block, and a non-string dtype (so a type error is a refusal, not a JSON exception escaping the loader). kv_int8() answers the target-capability question P4 will ask.
• tools/npu_plan.py — the generator states the contract it means (f16, no rotation) instead of leaving it to a key's absence. Nothing enables int8: that is a per-target decision whose rotation and scale sidecar live next to the kernels.
• tests/npu_plan_test.cpp — a case per rule above, including the two that matter for safety: ../elsewhere.signs is refused (a plan file must not point the loader outside the directories the command line named), and a non-string dtype is refused by name.
• tests/npu_kv_int8_test.cpp — the measured proof, below. Registered as ctest -R npu_kv_int8, header-only, no model and no device.
• docs/npu-plan.md — schema rows, refusal rules, a measured section, and P7's "Not yet" entry rewritten into what landed and what is left.

Measured proof

Both claims P7a rests on are arithmetic rather than device behaviour, so they are gated in CI. Head width 256 (what both models this phase targets carry), one scale per (head, token), 512 keys, 8 draws per cell; KLD of the attention weights against the f16 cache the plan calls the reference — the same statistic #806's gate names:

KV vectors	f16 + rotation	int8 K, unrotated	int8 K + rotation	rotation's worth
uniform (control)	3.9e-15	2.5e-05	2.4e-05	1.0x — nothing to concentrate, nothing claimed
8x outlier channels	8.6e-15	4.5e-04	4.7e-05	9.6x
30x outlier channels	4.5e-14	8.2e-03	1.5e-04	54.5x

1. The rotation is free. (QH)(KH)ᵀ = QKᵀ holds to 1.6e-09 relative, which is 0.01x one f32 dot product's own rounding error — a rotated f16 cache is not approximately today's cache, it is indistinguishable from it. If that were only approximately true, P7a would be a different model that answers the same text most of the time, which is the worst failure shape available.
2. The rotation is what keeps int8 inside the gate. At 30x outliers an unrotated int8 K cache lands at 8.2e-03 — an order of magnitude outside the 1e-3 gate — and the rotated one stays at 1.5e-04.
3. And the control says when it does not matter. At 8x outliers the unrotated cache is already inside 1e-3, so on a model with mild channels the rotation buys accuracy the gate does not ask for. That row is here so the PR does not claim a win on data where there is none; on uniform vectors the rotation measures 1.0x.

What this does not establish: the gate itself. The vectors are synthetic, generated to the outlier structure docs/npu-plan.md describes; #806's KLD is taken on real KV from a real model on the device, and that run is what remains.

Thresholds in the test are set from these columns, not chosen: the rotation's benefit must grow with outlier severity (54.5x > 9.6x > 1.0x), it must claim nothing on uniform data (<1.5x, measured 1.0x), and at 30x outliers the unrotated cache must be outside the gate while the rotated one is inside.

Checks

• ctest -R 'npu_kv_int8|npu_plan' — both green (plus the whole-tree target build via CMake, so the registration and the CI add_dependencies binding are exercised end to end, not just the standalone compile).
• tests/npu_plan_generate_test.py — green, so the generator change keeps every plan it writes valid.
• tools/copyright.py --check — rc 0; the new test carries the full Apache notice.
• No device: ~/GPU-TOKEN and the NPU keeper were never touched, and no ticket was taken.

What remains for #806 (also posted on the issue)

kv is a description until something reads it. In the order it can be taken: the cache writer/reader in the forward (int8 K store + scale sidecar, with the prefix-cache snapshot's byte accounting moving with it), the KLD gate on real KV on the device, then part B — prefill in 1024/2048-token chunks with block-fp expansion — which waits on P3's on-core expansion, P4's NpuModel, and the M>16 decision that part B needs and the decode path must not get.

[bong-water-water-bong]

#1273 ready for review at 94f80f8

Scoped first PR for #806 (plan P7), CPU-only: 0 s GPU, 0 s NPU, no tickets — rule 1 respected, c31 had the box.

Adds the kv contract to the plan (npu/plan/plan.h + generator + refusals) and gates the arithmetic part A rests on in CI with no device: (QH)(KH)ᵀ = QKᵀ to 1.6e-09 relative (0.01x one f32 dot's own rounding), and attention-weight KLD against f16 — uniform 1.0x, 8× outliers 9.6x, 30× outliers 54.5x, where the unrotated int8 cache is 8.2e-03 (outside the 1e-3 gate) and the rotated one is 1.5e-04 (inside). Control row included so nothing is claimed where there are no outliers.

ctest -R 'npu_kv_int8|npu_plan' 4/4 pass; npu_plan_generate green; copyright.py --check rc 0. Does not close #806 — the remaining list is in the issue comment.

[bong-water-water-bong]

/review

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 806 - Partially compliant Compliant requirements: The PR implements the KV cache contract in the plan layer with support for int8 KV and per-head Hadamard rotation It adds validation for KV cache data types and rotation parameters in the plan loader The PR includes CI tests that gate the arithmetic correctness of the rotation and int8 KV cache Backward compatibility is maintained with existing plans that don't specify KV cache details Non-compliant requirements: The PR does not implement the actual NPU prefill chunks with block-fp expansion (part B of P7) The PR does not implement the KLD gate on real KV from a model on the device Requires further human verification: Verification that the CI tests accurately reflect the performance claims in the PR description Confirmation that the arithmetic validation correctly bounds the mechanism as described

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing Validation for KV Value Type The plan loader validates that kv.key is either "f16" or "int8", but does not validate that kv.value is either "f16" or "int8". This could lead to undefined behavior if an invalid value type is specified for kv.value. if (p.kv_value != "f16" && p.kv_value != "int8") throw PlanError("plan: " + path + " kv.value \"" + p.kv_value + "\" is not f16 or int8"); Test Coverage for KV Value Type The test suite does not explicitly test the case where kv.value is set to "int8" and kv.key is "f16". This scenario should be tested to ensure that both key and value can be independently set to int8.