[bong-water-water-bong]

Why

scripts/npu-window11.sh --compare asserted that no prof row's ms/launch moves by more than a fixed 2 %, and
engine#834 showed that assertion cannot be evaluated on this box: the same binary against itself (8 alternating
rounds, sha256-identical, one NPU ticket) moved up to 7.7 %, the two obvious summary statistics disagreed in sign
on 5 of 9 rows, and the worst single round pair was off by ±33–43 %. Closes have meanwhile been citing "inside the
10 % tolerance" while the gate asserted 2 %, so the number in a close had no relation to what the gate checked.

What changed

The tolerance now comes from a measurement, or the comparison refuses to run.

scripts/npu-window11.sh --compare A.log B.log --tol PCT              # explicit, printed as such
scripts/npu-window11.sh --compare A.log B.log --control C1.log C2.log # same-binary pair sets it

With neither, --compare exits 2 and says why. With --control, the control's own worst row becomes the tolerance,
rounded up to the next whole percent (7.7 % → 8 %), and the control's per-row spread is printed so the number is
auditable.

The statistic is named. Every row prints both, and the header says which one the tolerance applies to:

statistic	meaning
ratio-of-medians	median(ms_B) / median(ms_A) — what the script printed before
median-of-ratios	median over rounds of ms_B / ms_A — paired; needs both logs to carry the same rounds

The paired one is used when both logs carry ≥2 rounds per row, because the box's noise is largely a level shift over a
round (someone else's traffic, a BO/ELF warm-up) that pairing cancels and a ratio of medians turns into a fake movement
— precisely the sign disagreement in the issue. A log with rounds carries the same prof/text lines once per round,
in round order.

A biased control is called what it is. If every control row moves the same way by a similar amount, the output says
it is a position bias — the second serve of a round paying for the warm-up — and that alternating A B then B A
inside a round is the fix, because no number of rounds averages a systematic bias out. That is the half of the issue the
script cannot do itself (the driver of the A/B pair owns the order), so it is stated in the output and in docs/npu.md.

Measured

The policy, end to end, device-free — scripts/npu-window11.sh --self-test, also registered as ctest
npu_window11_policy (a shell test needs no build target, so ci.yml's list is untouched):

ok   no tolerance and no control is refused
ok   --tol 10 passes a 5 % row and names the statistic
ok   --tol 10 still fails a 30 % row
ok   the control pair sets the tolerance from its own worst row (7.7 -> 8 %)
ok   a 30 % row still fails at the control's 8 %
ok   two rounds per row: the paired statistic is named and printed

self-test: PASS

On the real control logs from the issue's own session (~/p4-gate/out-ava-20261102T125310, same binary, 8 rounds —
tolerance derived from rounds 1–4, applied to rounds 5–8):

control: 9 rows from cA1.log vs cB1.log (same binary)
  dec D     M=128 ratio-of-medians  +28.0%  median-of-ratios  +26.6%
  ...
  pre QKV   M=128 ratio-of-medians  +15.5%  median-of-ratios  +12.2%
  note: every control row moves the same way (+12.2% .. +27.0%). That is a
        POSITION BIAS (the second serve of a round pays for the warm-up), not noise ...
statistic in force: median-of-ratios   tolerance 27% from control (worst row 27.0%, rounded up)
window11: PASS

Two things fall out of that run and are the reason the change is more than a flag: the control is not at 7.7 %
once the four rounds are paired (it is +12–27 %, all positive — the fixed ORDER="A B" bias), and the script now says
so instead of quietly recommending a 27 % tolerance as if it were noise. With the order alternated in the driver, that
systematic term disappears and the tolerance the control suggests drops to the noise floor.

python3 tools/copyright.py --check rc 0. No device claim is needed for this change; no NPU ticket was taken for it.

Closes #834.

Note on ownership: the issue's triage assigned it to pi-mosaic; the orchestrator reassigned it to this lane for the
overnight run. The alternating-order half lives in whatever drives the A/B pair, so if pi-mosaic's driver already does
it, the only thing to keep is the policy — and the --control number it prints is the one to quote in a close.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit d3b15a5)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 834 - Partially compliant Compliant requirements: The tolerance now comes from a measurement or explicit value The statistic is named in the output The script refuses to run without a tolerance The control's own worst row becomes the tolerance, rounded up Both statistics are printed for every row and the header names the relevant one Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 PR contains tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Control tolerance calculation The control tolerance calculation rounds up to the next whole percent, which may lead to a slightly higher tolerance than the actual worst row movement. While this is a conservative approach, it's important to ensure that this rounding does not mask real regressions. The rounding behavior should be verified against the intended policy. tol = float(max(der_tol, 1)) Self-test coverage The self-test script covers many cases but may not fully cover edge cases such as when the control logs have different numbers of rounds or when the paired statistic is not applicable. It would be beneficial to ensure that all possible combinations of inputs are tested to prevent unexpected behavior. if [ "${1:-}" = "--self-test" ]; then exec python3 - "$0" <<'PY' # The policy stays falsifiable without a device: write small logs, then check that a comparison with no # tolerance is refused, that an explicit tolerance is honoured and named, that a control pair sets the # tolerance from its own worst row, and that the paired statistic is used when the logs carry rounds. import os, subprocess, sys, tempfile script = sys.argv[1] d = tempfile.mkdtemp(prefix="window11-selftest-") failures = 0 def write(name, rows, texts=("a", "b", "c"), extra_rounds=0): p = os.path.join(d, name) with open(p, "w") as f: for _ in range(1 + extra_rounds): for i, t in enumerate(texts, 1): f.write(f"text {i} sha256={t} chars=1 text=x\n") for key, v in rows.items(): f.write(f"prof {key[0]} {key[1]} M={key[2]} ms={v:.3f} launches=100\n") return p keys = [("dec", "QKV", "16"), ("pre", "GU", "128")] base = {keys[0]: 2.000, keys[1]: 5.000} def run(*args): r = subprocess.run(["bash", script, "--compare", *args], capture_output=True, text=True) return r.returncode, r.stdout + r.stderr def check(ok, what, out=""): global failures print(("ok " if ok else "FAIL ") + what) if not ok: failures += 1 print(" " + out.replace("\n", "\n ")[:400]) a = write("a.log", base) b = write("b.log", {keys[0]: 2.100, keys[1]: 5.150}) # +5 % and +3 % c1 = write("c1.log", base) c2 = write("c2.log", {keys[0]: 2.154, keys[1]: 5.050}) # same binary: worst row 7.7 % bad = write("bad.log", {keys[0]: 2.600, keys[1]: 5.100}) # +30 % and +2 % rc, out = run(a, b) check(rc == 2 and "no tolerance given" in out, "no tolerance and no control is refused", out) rc, out = run(a, b, "--tol", "10") check(rc == 0 and "statistic in force: ratio-of-medians" in out and "tolerance 10% from --tol" in out, "--tol 10 passes a 5 % row and names the statistic", out) rc, out = run(a, bad, "--tol", "10") check(rc == 1 and "outside tolerance" in out, "--tol 10 still fails a 30 % row", out) rc, out = run(a, b, "--control", c1, c2) check(rc == 0 and "tolerance 8% from control" in out and "worst row 7.7%" in out, "the control pair sets the tolerance from its own worst row (7.7 -> 8 %)", out) rc, out = run(a, bad, "--control", c1, c2) check(rc == 1, "a 30 % row still fails at the control's 8 %", out) # two rounds per arm whose per-round ratios (0.90, 1.20) are not the ratio of medians pa = os.path.join(d, "pa.log") with open(pa, "w") as f: for r in range(2): for i, t in enumerate(("a", "b", "c"), 1): f.write(f"text {i} sha256={t} chars=1 text=x\n") for key in keys: f.write(f"prof {key[0]} {key[1]} M={key[2]} ms={base[key]:.3f} launches=100\n") pb = os.path.join(d, "pb.log") with open(pb, "w") as f: for r, fac in enumerate((0.90, 1.20)): for i, t in enumerate(("a", "b", "c"), 1): f.write(f"text {i} sha256={t} chars=1 text=x\n") for key in keys: f.write(f"prof {key[0]} {key[1]} M={key[2]} ms={base[key] * fac:.3f} launches=100\n") rc, out = run(pa, pb, "--tol", "10") check("statistic in force: median-of-ratios" in out and "median-of-ratios" in out, "two rounds per row: the paired statistic is named and printed", out) print() print("self-test: " + ("PASS" if not failures else "FAIL")) sys.exit(1 if failures else 0) PY fi

[bong-water-water-bong]

VERDICT: READY

Closes #834 properly: the tolerance is no longer a constant the script can pretend to check, and I exercised all six
behaviours with my own fixture logs rather than through its self-test (which also passes, 6/6).

fixture	what the script did
--compare c1 c2 with no --tol/--control	exit 2, message quoting the #834 measurement ("the SAME binary against itself moving up to 7.7 % per row") and naming both acceptable inputs
--control c1 c2 (worst row 7.7 %)	prints the control's per-row spread (+7.7 / +1.0 / -1.0), then tolerance 8% from control (worst row 7.7%, rounded up) and statistic in force: ratio-of-medians
control whose rows all move the same way (+6.0 % each)	prints the POSITION BIAS note — "not noise, and more rounds will not average it out: alternate the order inside a round" — and still derives the tolerance
two rounds per row in both logs	statistic in force: median-of-ratios, with both columns printed per row
a 30 % row against the control's 8 %	exit 1 (outside tolerance)
same prof rows, one prompt's text differs	exit 1 — so the transcript half still gates when the tolerance half passes

The two things I most wanted to see are both real: the refusal has no silent default (grep for a 2 % constant finds only
the comment explaining why it cannot exist), and the paired statistic is actually selected by the data shape rather than
announced. add_test(NAME npu_window11_policy … --self-test) is registered in CMakeLists.txt, so CI runs the policy
check itself, and it needs no build target (the comment says so, which is the trap this repo has hit three times).

docs/npu.md says the same thing the code does, including the half the script cannot do for you — "alternate the order
inside a round" — which is the honest split, since the A/B driver owns that.

One note for the next pass rather than a gate: the position-bias detector needs ≥3 rows and sd/|mean| < 0.5; a control
with two rows, or with one clean row and two biased ones, will not trip it and will report a tolerance inflated by the
warm-up. Nothing in the output hides that (the per-row spread is printed), but the note is the only thing that turns it into
an explanation.

[github-actions]

Persistent review updated to latest commit e9e864b

[bong-water-water-bong]

VERDICT: READY

At e9e864b5d (merge of main into the head I reviewed; both files blob-identical, so the content is the content I verified).
Build success at this head.

Evidence: exercised all six behaviours with my own fixture logs, not through the self-test — no --tol/--control → exit 2 quoting the #834 numbers; --control → per-row spread plus tolerance 8% from control (worst row 7.7%, rounded up); a uniform +6 % control → the POSITION BIAS note; two rounds per row → statistic in force: median-of-ratios; a 30 % row → exit 1; a text-only mismatch → exit 1. No silent 2 % default remains, and npu_window11_policy is a registered ctest.

[github-actions]

Persistent review updated to latest commit fbedbd0

[bong-water-water-bong]

VERDICT: READY

At fbedbd072 (merge of main into the head I reviewed; 3 of 3 files blob-identical, so the content is what I verified).
Build success at this head.

Evidence stands: all six behaviours exercised with my own fixtures — refusal with no --tol/--control (exit 2, #834 numbers in the message), control-derived tolerance 8% from a 7.7 % worst row, the POSITION BIAS note on a uniform control, median-of-ratios chosen when both logs carry rounds, exit 1 for a 30 % row and for a text-only mismatch; npu_window11_policy is ctest-registered.

[github-actions]

Persistent review updated to latest commit d3b15a5