bong-water-water-bong · Nov 6, 2026 · Nov 5, 2026 · Nov 6, 2026
diff --git a/docs/onnx-pack.md b/docs/onnx-pack.md
@@ -216,35 +216,45 @@ intermediate float tensor then never exists.
   down_proj's ship beside the other xclbins, and q/k/v's and gate/up's came out of one `-M 128 -K <K>
   -N <N> -c 4` run of the iso-build generator each (`n1_core_i8_v26.py` through the IRON/MLIR-AIE
   `aiecc`), the artifacts living beside the other xclbins and never in this repository. It is a
-  **correctness** result — and at model level it is **not yet a speed one**, which is now measured rather
-  than suspected. Same export, a Release build of this tree, a quiet box (load 1.2–1.4), one thread, a
+  **correctness** result, and at model level it is a **small** speed one — 8.3 % on a quiet box on the
+  current tree, after the same export measured 1.13–1.16× *slower* on an earlier one; both are measured
+  rather than suspected, and the last row below is the number to quote. Same export, a Release build of
+  this tree, a quiet box (load 1.2–1.4), one thread, a
   32-token prompt and 8 generated tokens: the array's arm does **1074 / 1105 ms/token against the host
   kernel's 947 / 951**, i.e. 1.13–1.16× *slower* than the leg it replaces (and 1.7× behind the A8W8 CPU EP
   at 625 / 628, 4.8× behind float32 at 226); the 32-token prefill is a wash, 805 / 809 against 843 / 845.
   **Re-measured on the current tree those two arms are 930.5 against 972.9 ms/token, i.e. the array 4.4 %
   ahead** — same window, two runs each, both agreeing on the sign, and the profile now splits a pass into
   device calls 802.5 + epilogue 19.0 against the host loop's 879.0 with 94–109 ms of per-pass work that
-  neither leg touches. Not the code, as far as the repository can say: since those figures, only #1032
+  neither leg touches. **Repeated on a quiet box (load1 1.3–3.3) the same sign is larger and is now the
+  quotable one: 983.4 / 1000.2 against 1073.6 / 1088.8 ms/token, i.e. 8.3 % ahead**, per call 4.32 against
+  the host loop's 4.99. Not the code, as far as the repository can say: since those figures, only #1032
   (profiling) and #1047 (the re-pack, ~5 % of a token) touch this path, the NPU runtime the EP links
-  against is unchanged and the five ELFs are byte-identical — so ~5 % of the 13 % is code and the rest is
-  unestablished, most plausibly CPU-side contention in the earlier session, since half of a device call is
-  host work. So the sign is no longer the finding — the magnitude is, and it is small in both directions;
-  quote the number only with the session it came from (`docs/onnx.md`, "Where a pass actually goes").
+  against is unchanged and the five ELFs are byte-identical, so the 1.13× loss does not reproduce: the host arm
+  of the window above ran at load1 1.3–1.6, the same load as the 1.2–1.4 that loss was taken at, and there the
+  array is ahead — though that window is not an arm-for-arm repeat of it, since its device arm ran at 2.8–3.3.
+  So the sign is no longer the finding — the magnitude is,
+  and it is small in every window; quote the number only with the session it came from (`docs/onnx.md`,
+  "Where a pass actually goes").
   The kernels are ahead **in isolation** — 1.54–1.89× per call at a decode step's 36 rows and 3.1× at a
   32-row prefill block — and that advantage does not survive the model. These measurements do not, by
   themselves, say which of the two candidate reasons it is: non-kernel work per call (BO writes, two syncs,
   a wait, the weight writeback, the shared epilogue); or the host loop being faster on the graph's real
   operands than on the probe's synthetic ones (`docs/onnx.md` explains why the second cannot be excluded).
-  Three of those per-call costs are since measured on the array — weight writeback 0.24 ms, epilogue
-  0.134 ms, pack 0.06 ms, i.e. 0.43 ms/call or 84 ms of a 1074 ms pass — and the weight path is not worth
-  fixing: keeping every image resident returns 35 ms/pass for 568 MiB (`docs/onnx.md` carries both). The
-  second candidate turned out to be measurable without the array: dumped from one prefill, the graph's real
+  Three of those per-call costs were measured separately on the array (weight writeback 0.24 ms, epilogue
+  0.134 ms, pack 0.06 ms), and the weight path is not worth fixing: keeping every image resident returns
+  35 ms/pass for 568 MiB (`docs/onnx.md` carries both). The call's own sections then priced all of them in
+  one window: of a 4.32 ms call the launch plus its wait is 3.90 (90 %, and that section *contains* the
+  kernel's multiply), the weight pack plus upload 0.25, `other` 0.13, the readback 0.036, both flushes 0.008
+  — so every non-multiply thing the leg does per call is **0.42 ms (9.8 %)**, 0.54 with the 0.12 epilogue it
+  pays outside the call, and no fix aimed at them can recover more.
+  The second candidate turned out to be measurable without the array: dumped from one prefill, the graph's real
   int8 activations are 62.1 % nonzero against the probe's ~99 %, which makes its host loop **1.42× faster**
   per pass (1098 → 770.6 ms) and cuts the array's kernel advantage from 1.7× to **~1.12×**. So the multiply
-  is not where the model-level loss comes from, and what is left is the per-call work inside `dev_call_ms`
-  that the probe does not see — which the both-arms profile then measured: 4.09 ms/call at model level against
-  the probe's 3.460 for a single shape in the same window, i.e. the EP adds no structural per-call cost, and
-  the arms finished 930.5 against 972.9 ms/token in the array's favour. (Those arms, like every
+  is not where the *1.13× loss* of the figure above came from, and that loss does not reproduce on the
+  current tree at any load tried: what is there is the 8.3 % win above, whose per-call cost the probe also
+  prices (4.32 ms/call at model level against its 3.460 for a single shape in the same window, i.e. the EP
+  adds no structural per-call cost). (Those arms, like every
   number on this page, come from a Release build: the EP targets carry no `-O2` of their own, and an
   unoptimized host loop inflates the device-against-host ratio by ~25×.)
   `docs/onnx.md` ("What the device leg is worth at model level") carries the four-arm table, the per-shape

diff --git a/docs/onnx.md b/docs/onnx.md
@@ -261,9 +261,9 @@ its KL against float is the same number (5.6787 nats both; the A8W8 CPU EP's own
 different continuation), and its trace is `qdq runs=1764` (= 196 groups × 9 passes), `host-rows policy=0`,
 `device fallbacks=0`, `device-busy=0` — every fused group went to the array, and nothing was dressed up as one.
 **These are the figures as published then, on a tree before #1047; the same two arms re-measured on the current tree
-are 930.5 against 972.9 ms/token in the device's favour — see "Where a pass actually goes, and the sign on the current
-tree" at the end of this section, which is the number to quote, and which explains the difference rather than
-overwriting it.**
+are 983.4–1000.2 against 1073.6–1088.8 ms/token in the device's favour on a quiet box (and 930.5 against 972.9 at
+load 4.9–6.9 before it) — see "Where a pass actually goes, and the sign on the current tree" at the end of this
+section, which is the number to quote, and which explains the difference rather than overwriting it.**
 The 5-token counter-example is the other half of that claim: same binary, same designs, 8 tokens in 2159 ms
 (3.70 tok/s), trace `host-rows policy=1764` and no device call at all, because a 5-token prompt never reaches 16
 rows.
@@ -371,39 +371,70 @@ back) and `dev_split_other` (the device lock, the kernel lookup and the EP's own
 five named sections do not account for) — so `dev_call_ms` adds up on the face of the line rather than leaving a reader
 to attribute one residue by subtraction. `NPU_C_SYNC_PROF=1` (engine#988) still prints the C-sync counters per call
 if a section needs separating further. These sections are timed only when the knob is on (a plain call pays one branch
-per section), and **no device window has printed them yet** — they appear from the next one on. What a *fix* would
+per section), and they have now printed once: on a quiet box the launch plus its wait is **3.90 of a 4.32 ms call**
+(90 %), the weight pack 0.25, the readback 0.036 and the epilogue 0.12 — the table below. So (1) is where the per-call
+cost is. What a *fix* would
 look like if (1) dominates is one launch covering many projections — a resident weight layout and a pass-level submission, rather
 than the per-projection ELF this leg uses — which is also why the threshold discussion above stands unchanged:
 `kDeviceMinRows` is about one call, and the model-level question is about 196 of them. Worth knowing before anyone
 starts that work: the resident-weight half of it has now been measured and is worth 35 ms/pass for 568 MiB, i.e. the
 sign of the model-level result does **not** turn on the weight path.
 
 **Where a pass actually goes, and the sign on the current tree.** The same two EP arms, two runs each, back to back in
-one `npu-hold` ticket (load 4.9–6.9, identical ids, `fallbacks=0`, `device-busy=0`), with `ONEBIT_EP_QDQ_PROF=1` on
-both:
+one `npu-hold` ticket with `ONEBIT_EP_QDQ_PROF=1` on both — this is the quiet-box repetition the rows above asked for,
+load1 1.3 and 1.6 for the host arm and 3.3 and 2.8 for the device arm (the gate is load1 < 4; the 15-minute average was
+still falling from 9.8, this lane's own build), identical ids, `fallbacks=0`, `device-busy=0`, and all 1764 fused calls
+= 196 groups × 9 passes on the array:
 
 | arm | decode | per pass, from the profile |
 |---|---|---|
-| our EP, **array** | 953.3 / **930.5** ms/token | device calls 802.5 (`dev_call_ms` 7222.2 / 9), epilogue 19.0 (`dev_epi_ms` 171.2 / 9), other 109 |
-| our EP, host kernel | 970.4 / **972.9** ms/token | host loop 879.0 (`host_ms` 7911.2 / 9), other 94 |
-
-Both arms are fully accounted for (802.5 + 19.0 + 109 = 930.5; 879.0 + 93.9 = 972.9), and "other" — attention, norms,
-softmax, embed, ORT — is 94–109 ms/pass, the part neither leg touches. So on this tree the device arm is **4.4 % ahead**,
-not 1.13× behind as the four-arm table far above reports, and both runs of each arm agree on the sign. The host
-loop's 879 ms/pass also validates the CPU-only estimate of 848 from the section above to within 4 %.
-
-What differs between those two sessions is **not** the code, as far as the repository can say: since the build those
+| our EP, **array** | 1000.2 / **983.4** ms/token | device calls 846.5 (`dev_call_ms` 7618.2 / 9), epilogue 23.6 (`dev_epi_ms` 212.1 / 9), other 113.4 |
+| our EP, host kernel | 1073.6 / **1088.8** ms/token | host loop 978.9 (`host_ms` 8810.2 / 9), other 109.9 |
+
+Both arms are accounted for (846.5 + 23.6 + 113.4 = 983.4; 978.9 + 109.9 = 1088.8), and "other" — attention, norms,
+softmax, embed, ORT — is the part neither leg touches. It is also the noisiest row: 54 and 110 ms/pass on the host's two
+runs, 119 and 113 on the device's, so a one-minute window's non-EP work moves by ~50 ms. That does not weaken the sign —
+the 89 ms gap between the arms' means is larger than the spread on the row that is supposed to be identical — but it is
+why the claim is the mean of two runs and not one. At the load the claim needs the array is
+**8.3 % ahead** (mean of the two runs, 991.8 against 1081.2 ms/token; per call 4.32 against the host loop's 4.99), and the
+32-token prefill is ahead by more, 725.5 / 746.3 ms against 1409.1 / 878.9 — those two host numbers are 60 % apart, so
+prefill is "ahead, magnitude unsettled" rather than a ratio to quote. Two earlier windows reported 1.13–1.16× *behind*
+at load 1.2–1.4 and 4.4 % ahead at load 4.9–6.9; the sign is now measured at the load the claim needs, and its magnitude
+is small in every window.
+
+**The call's own sections, first printing.** Per call over 1764 calls (run 2), against `dev_call_ms` = 4.32:
+
+| section | ms/call | share |
+|---|---|---|
+| `dev_split_launch` (launch + `wait_kernel`) | **3.90** | 90 % |
+| `dev_split_wpack` (weight pack + upload, a cache miss) | 0.25 | 5.8 % |
+| `dev_split_other` (device lock, kernel lookup, EP frame) | 0.13 | 3 % |
+| `dev_split_readback` (C invalidate + copy back) | 0.036 | 0.8 % |
+| `dev_split_aupload` (A flush) | 0.004 | 0.1 % |
+| `dev_split_async` | 0.004 | 0.1 % |
+
+Read that against the correction this file's history already carries, not as a headline: `launch` + wait **contains the
+kernel's own run time**, so 3.90 is not dispatch overhead waiting to be deleted — the probe prices q at m=36 at 3.460
+ms/call device against 5.836 host, i.e. most of that section *is* the multiply. What the split does establish, by
+subtraction, is that every other thing the leg does per call is **0.42 ms combined (9.8 % of `dev_call_ms`)** — the
+weight path 0.25, `other` 0.13, the readback 0.036, both flushes 0.008 — plus the 0.12 epilogue the device arm pays
+outside it; the weight path figure is corroborated independently (0.24 ms via #1047). Those are not where the
+model-level advantage leaks, and no fix aimed at them can recover more than that. The remaining lever is the shape of the
+submission — 196 per-projection ELF calls per pass — and its dispatch residue is what is left of that section once the
+multiply the probe prices (3.460 ms/call for one shape in the same window) is taken out of the 4.32.
+
+What differs between those sessions is **not** the code, as far as the repository can say: since the build the 1.13×
 figures came from, exactly two commits touch this path — #1032, which adds the profiling that produced the numbers above,
 and #1047, the weight re-pack, 0.24 ms/call = 47 ms/pass (~5 % of a token) — the private NPU runtime the EP links against
-is unchanged (files dated 2026-10-31), and the five design ELFs are byte-identical (md5s). It is not load in the obvious
-direction either: this run was at load 4.9-6.9 against 1.2-1.35 then, and the device arm gets *slower* under load (its
-calls measured 5.65 ms/call at load 11-14 against 4.09 here). So ~5 % of the 13 % is code and **the rest is
-unestablished**: the plausible shape of it is CPU-side contention in the earlier session that a one-minute load average
-did not capture, since half of a device call is host work (BO writes, two syncs, the readback), and the honest reading is
-that those are two sessions under different conditions rather than two builds. The kernel is unchanged either way — the
-probe in this same window still measures q at m=36 at 3.460 ms/call device against 5.836 host, the 1.69x the rows table
-reports. What moved is the model-level per-call cost, and a quiet-box repetition is what this number needs
-before anyone quotes it as the device leg's worth.
+is unchanged (files dated 2026-10-31), and the five design ELFs are byte-identical (md5s). Nor is the tree: the host arm
+of the window above ran at load1 1.3–1.6, the same load as the 1.2–1.4 the loss was taken at, and there the array is
+ahead — so the 1.13× loss does not reproduce at that load, although this window is not an arm-for-arm repeat of it
+(its device arm ran at 2.8–3.3), and what produced it in that session this lane cannot say: its build lived in `/tmp`,
+so it no longer exists to run beside this one. The
+kernel is unchanged either way — the probe in the same window still measures q at m=36 at 3.460 ms/call device against
+5.836 host, the 1.69× the rows table reports. So: the per-call multiply advantage is real, the model-level advantage is
+small and now measured on a quiet box, and the per-call non-multiply work is priced at 0.42 ms (0.54 with the epilogue)
+rather than guessed at.
 
 **The device lock.** The NPU is shared, and the rule is that a plugin never evicts the owner's
 serve. The EP takes `/tmp/1bit-npu-device.lock` with `flock(2)` **non-blocking** and holds it for the