diff --git a/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json b/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json index 9ef228af..89172ff2 100644 --- a/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json +++ b/bench/npu-leaderboard/bonsai2-27b-pb16i-20261102.json @@ -17,8 +17,8 @@ "stream_per_token_gb": 12.93, "tokens": 7 }, - "instrument": "NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", - "repeats": 7, + "instrument": "NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", + "repeats": 1, "timestamp": "2026-11-02T23:50:00Z", "source": "~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 409-412 ms/token", "notes": "control arm of the P3 (#802) pair: the same container file, the same binary and the same box one minute from bonsai2-27b-t2s2-20261102.json, so the ratio is a weight-format difference and nothing else. This is the 4.5 bpw path the 2-bit stream replaced; it is not a candidate, it is the denominator. What would make this wrong: measuring it in a different window than the t2s arm (the pair is only valid adjacent)." diff --git a/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json b/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json index 65e15b55..5a3b73ce 100644 --- a/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json +++ b/bench/npu-leaderboard/bonsai2-27b-t2s2-20261102.json @@ -17,9 +17,9 @@ "stream_per_token_gb": 6.11, "tokens": 7 }, - "instrument": "NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", - "repeats": 7, + "instrument": "NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once", + "repeats": 1, "timestamp": "2026-11-02T23:50:00Z", "source": "~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token", - "notes": "P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept: every decode role byte-identical to the pb16i path (6/6 at layer 0, 8/8 at layer 3, seeds 1/2/3) and greedy text identical on the three-prompt window. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control." + "notes": "P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept is byte-exactness, not a text diff: every decode role matched the pb16i path byte for byte at layer 0 (6/6) and layer 3 (8/8) on three seeds, and greedy text is equal to the pb16i arm's on the prompt both were asked (' Paris.\\nThe capital of Germany is'). A three-prompt full-model text window on this pair was NOT run - the 2026-11-06 re-take was blocked by an unrelated serve holding the engine's device lock - so that remains the open acceptance step. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control." } diff --git a/docs/npu-leaderboard.md b/docs/npu-leaderboard.md index 631aa22a..92f9a254 100644 --- a/docs/npu-leaderboard.md +++ b/docs/npu-leaderboard.md @@ -24,8 +24,8 @@ One row is one measured run of this engine. A record is only ranked if it names | model | route | quant | metric | value | settings | repeats | instrument | | --- | --- | --- | --- | --- | --- | --- | --- | -| Bonsai-2-27B | npu | PTQ1_0 ternary, t2s 2-bit stream (2.125 bpw) | decode | 313.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-t2s2 (q4nx_weight_layout=t2s, version 6, sealed SEALED-T2S.elf, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=6.11, tokens=7 | 7 | NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | -| Bonsai-2-27B | npu | pb16i (int4 codes + int16 magnitudes, 4.5 bpw) | decode | 410.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-pb16i (q4nx_weight_layout=pb16i, version 4, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=12.93, tokens=7 | 7 | NPU_PHASE_PROF=1, the serve's own [ph] tok wall= line, median of the 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | +| Bonsai-2-27B | npu | PTQ1_0 ternary, t2s 2-bit stream (2.125 bpw) | decode | 313.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-t2s2 (q4nx_weight_layout=t2s, version 6, sealed SEALED-T2S.elf, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=6.11, tokens=7 | 1 | NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | +| Bonsai-2-27B | npu | pb16i (int4 codes + int16 magnitudes, 4.5 bpw) | decode | 410.0 ms/token | device=accel0, port=18099, container=~/models/npu/bonsai2-27b-pb16i (q4nx_weight_layout=pb16i, version 4, M=16), binary=~/wt/bonsai-speed-build2/1bit (2026-11-02, engine 916e90d + the Hadamard SIMD host path), stream_per_token_gb=12.93, tokens=7 | 1 | NPU_PHASE_PROF=1, one run per arm: the serve's own [ph] tok wall= line, median of its 7 steady-state tokens (first token dropped); request walls are not this number, they pay the per-serve image fill once | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill pp512 | 286.63 tok/s | ngl=99, fa=1, ub=512, engine=0d0c846 | 3 | llama-bench -p 512, 3 repeats, s.d. 0.94 tok/s | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill p1024 ub576 | 269.34 tok/s | ngl=99, fa=1, ub=576, engine=0d0c846 | 1 | llama-bench -p 1024, 1 repeat | | Flash-Next (IQ4_XS 4.25 bpw) | hrx | IQ4_XS | prefill pp4096 | 260.59 tok/s | ngl=99, fa=1, ub=512, engine=0d0c846 | 3 | llama-bench -p 4096, 3 repeats, s.d. 4.51 tok/s | @@ -46,7 +46,7 @@ One row is one measured run of this engine. A record is only ranked if it names | Qwen3.8-27B-NPU2 | npu | Q4NX (int8 GEMM) | first token (weight pack included) | 145.7 s | device=accel0, commits=800cf6f, threads=32 | 1 | request wall and driver trace span | | Qwen3.8-27B-NPU2 | npu | Q4NX (int8 GEMM) | first token (weight pack included) | 435.9 s | device=accel0, commits=pre-#622, threads=32 | 1 | driver trace span, gated on ~/npu.lock | -- `bonsai2-27b-t2s2-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token; P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept: every decode role byte-identical to the pb16i path (6/6 at layer 0, 8/8 at layer 3, seeds 1/2/3) and greedy text identical on the three-prompt window. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control. +- `bonsai2-27b-t2s2-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 311-317 ms/token; P3 (#802) 'after' arm against bonsai2-27b-pb16i-20261102.json: 1.31x per token and 1.42x on the device wait (209 vs 296 ms). Accept is byte-exactness, not a text diff: every decode role matched the pb16i path byte for byte at layer 0 (6/6) and layer 3 (8/8) on three seeds, and greedy text is equal to the pb16i arm's on the prompt both were asked (' Paris.\nThe capital of Germany is'). A three-prompt full-model text window on this pair was NOT run - the 2026-11-06 re-take was blocked by an unrelated serve holding the engine's device lock - so that remains the open acceptance step. Stream is 2.12x fewer bytes (431 images, 14.44 -> 6.82 GB). What would make this wrong: quoting a request wall instead of [ph] tok (a fresh serve adds the ~10.6 s weight-image fill), or comparing across boxes - the pb16i arm is the only valid control. - `bonsai2-27b-pb16i-20261102` — 2026-11-02T23:50:00Z; source: ~/briefs/TBONSAI-SPEED-STATE.md, 2026-11-02 20:05 AST entry: three npu-hold tickets (19:47, 19:48, 19:49 AST) on :18099, amdxdna counter 0 in each, spread 409-412 ms/token; control arm of the P3 (#802) pair: the same container file, the same binary and the same box one minute from bonsai2-27b-t2s2-20261102.json, so the ratio is a weight-format difference and nothing else. This is the 4.5 bpw path the 2-bit stream replaced; it is not a candidate, it is the denominator. What would make this wrong: measuring it in a different window than the t2s arm (the pair is only valid adjacent). - `flashnext-pp512-20261027` — 2026-10-27T02:38:52Z; source: ~/lb/572/run-20261027T015408Z/a1-timing.out (strixhalo); gufo reference on the same box is 1679 tok/s; the analysis is in docs/hrx.md - `flashnext-p1024-ub576-20261027` — 2026-10-27T02:50:00Z; source: ~/lb/572/run-20261027T015408Z/a4-ub576.out (strixhalo); shows the ubatch bound above 512 no longer refuses the run; single repeat, do not read as a tuned number