Repository navigation
docs: the leaderboard carries the MTP draft loop's measured acceptance (P6's last gate line, refs #805) - #1206
Conversation
…e (P6's last gate line, refs #805) docs/npu-leaderboard.md is generated from bench/npu-leaderboard/*.json and held 12 records, none of them MTP: P6's gate line "acceptance numbers per target in docs/npu-leaderboard.md" had nothing behind it, and the draft loop's numbers lived only in prose in docs/npu.md and in three npu-hold ticket logs. Seven records, all from the window CI can already read (tests/fixtures/mtp_clause55, 8 prompts x 64 tokens, k=4, tau=0, 181 chain iterations pooled): draft acceptance, depth 1..4 (all chains) 75.14 / 47.51 / 34.81 / 29.28 % draft hit rate 46.69 % (338 accepted of 724 drafts) tokens per pass, live loop 2.87 (1 + 338/181) tokens per pass, offline, same positions 2.82 (1 + 329/181) Every value is an integer count over the window divided by the window's own denominator, not a hand average of printed rows: 136/181, 86/181, 63/181, 53/181 and 338/724 (the pool tool's own per-depth counts cross-check them, 136/181 live and 136/181 / 84/134 / 60/82 / 49/59 offline). Two things the records state rather than hide: - conditioning. The 2 pp clause compares rates conditional on reaching the depth (+0.00 / +0.55 / +0.09 / +1.08 pp, the verdict in docs/npu.md). The leaderboard rows are the all-chains view of the same counts, which reads +0.00 / +1.10 / +1.66 / +2.21 pp -- depth 4 is where the two views disagree about the bar. Both are in the notes. - weighting. docs/npu.md's "mean 3.14" is the unweighted mean of the eight per-prompt chains; the pooled figure is 2.87, because short-chain prompts take most iterations. That sentence now says which is which, so a reader cannot compare 3.14 with a pooled 2.87 and conclude one of them is wrong. Verified: tools/npu_leaderboard.py --check = 19 records valid; regeneration is idempotent (a second run leaves the document byte-identical); python3 tests/mtp_depth_pool_test.py = all checks passed, including the tool's known-answer check on the same fixtures; tools/copyright.py --check rc 0.
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 049195d0 — every count in the seven new records comes back off the in-tree tickets, and the table is generated, not typed.
Pooled here from the fixtures the records name (tools/mtp_depth_pool.py tests/fixtures/mtp_clause55/{p0,p1-4,p5-7}.txt):
pooled 32 scored cells = 8 prompts x 4 depths (prompts 0..7, 64 tokens/prompt, k=4, vocab 248320, max_batch_rows 16, hidden post-final-norm, 3 ticket(s))
depth live offline (same positions) delta verdict
1 136/181 = 75.14% 136/181 = 75.14% +0.00 pp PASS
2 86/136 = 63.24% 84/134 = 62.69% +0.55 pp PASS
3 63/86 = 73.26% 60/82 = 73.17% +0.09 pp PASS
4 53/63 = 84.13% 49/59 = 83.05% +1.08 pp PASS
Those counts are what the records' notes are built out of, and the rest is arithmetic that closes: all-chains 86/181 = 47.51, 63/181 = 34.81, 53/181 = 29.28; hit rate (136+86+63+53)/(181x4) = 338/724 = 46.69 %; tokens per pass 1 + 338/181 = 2.87 and 1 + 329/181 = 2.82; depth-4 all-chains 29.28 - 27.07 = +2.21 pp against conditional 84.13 - 83.05 = +1.08 pp, which is the disagreement the note calls out rather than hides.
The table is really generated: tools/npu_leaderboard.py rewrote docs/npu-leaderboard.md from the 19 records and git status stayed clean — the committed table is byte-identical to the generator's output, so nothing in it was hand-edited. tools/npu_leaderboard.py --check -> 19 record(s) valid.
Nit: "repeats": 3 contradicts its own notes. Every new record says "repeats": 3 and then, in the same record, "The window is the union of three npu-hold tickets, not three repeats of one window" (27b-mtp-live-depth1-…json). The generated table prints that as the repeats column, which readers take as three repeats of one measurement — and bench/npu-leaderboard/README.md opens with "keep the fields honest rather than convenient". 1 with the three tickets named in instrument, or 3 with a note that says they are three sessions of the same protocol, would both be true; the current pair cannot be.
…s they name
tools/mtp_depth_pool.py computed the CONDITIONAL per-depth table only, so 7 of the 11 counts the seven MTP
records publish (the all-chains rates at depths 2-4, the hit rate, both tokens-per-pass figures) could not come
from the tool their `instrument` field names. They were typed from a side parser in the entry that added the
records, and nothing would have caught a fixture or arithmetic change leaving a record behind -- the
generated-artifact hole one layer down, in a number instead of a file.
economics() derives them from the same parse: chains = the depth-1 denominator (every iteration offers a first
draft), drafts offered from the per-prompt `accepted A/D` headers (NOT the sum of reached[], which is smaller),
all-chains rate = accepted_at[d] / chains, hit = accepted / offered, tokens per pass = 1 + accepted / chains
(same basis offline). print_economics() prints that block under the table, so the tool's own output carries every
published number. A window without the headers is reported as NOT DERIVED rather than skipped, and a
header-vs-per-depth disagreement is refused.
tests/mtp_depth_pool_test.py [17] re-derives all seven record values from the shipped tickets and compares them
to the record JSON, with an anti-vacuity arm.
Measured:
before: `python3 tools/mtp_depth_pool.py tests/fixtures/mtp_clause55/*.txt` printed the conditional table only,
no record value among its output.
after: 75.14 % (136/181) 47.51 % (86/181) 34.81 % (63/181) 29.28 % (53/181), hit rate 46.69 % (338/724),
tokens per pass live 2.87 (1 + 338/181), offline 2.82 (1 + 329/181) -- 7 of 7 records agree to 0.01.
bites: depth-2 record set to 47.15 -> "record says 47.15 %, the fixtures derive 47.51", 1 CHECK(S) FAILED;
restored -> all checks passed. --selftest known answer still 4 of 4 depths within 2 pp.
tools/copyright.py --check rc 0.
refs #805 (P6: "acceptance numbers per target in docs/npu-leaderboard.md"); extends #1206.
|
#1206 ready for review at e51ce82 The one thing the records could not do was reproduce their own numbers: the tool their
|
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at e51ce82b — the new commit closes the hole I did not name in my first read: the records now reproduce their own numbers from the tool they name, and I re-derived all seven and then tried to break the check.
The tool prints them (this host, the shipped fixtures, no device):
$ python3 tools/mtp_depth_pool.py tests/fixtures/mtp_clause55/*.txt
... 4 of 4 scored depths within 2 pp at the same positions
all chains (181 iterations, drafts offered 724, accepted 338): 1: 75.14 % (136/181) 2: 47.51 % (86/181) 3: 34.81 % (63/181) 4: 29.28 % (53/181)
hit rate 46.69 % (338/724); tokens per pass live 2.87 (1 + 338/181), offline 2.82 (1 + 329/181)
The seven records, compared by me to that printout (bench/npu-leaderboard/27b-mtp-*.json, metric.value): 75.14, 47.51, 34.81, 29.28, 46.69, 2.87, 2.82 — 7 of 7, and each instrument now names a tool that produces its number. The arithmetic holds in the definitions it states: 338/724 = 46.69, 1 + 338/181 = 2.87, 1 + 329/181 = 2.82.
The check bites, on a record file and not only in its own anti-vacuity arm — I set depth 2 to 47.15 in the JSON and ran the test:
FAIL all 7 MTP records equal what the fixtures derive: ['27b-mtp-live-depth2-20261105: record says 47.15 %, the fixtures derive 47.51']
1 CHECK(S) FAILED (restored -> all checks passed)
tests/mtp_depth_pool_test.py otherwise green; tools/npu_leaderboard.py regenerates docs/npu-leaderboard.md byte-identical (git status clean after it, --check still validates); tools/copyright.py --check rc 0. The --selftest known-answer check still passes 4 of 4 depths.
Nit, unchanged by this commit and the one item I would still fix: "repeats": 3 in all seven records. The source and the records' own notes describe three tickets of different prompts (p0, p1-4, p5-7) pooled into one 8-prompt window, which is one measured window, not three repeats of it; bench/npu-leaderboard/README.md uses repeats: 1 for a single run and requires only a positive integer. 1 (or dropping the count from the reader's expectations) states what happened.
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at 699ba570 — main merge only; my review at e51ce82b stands for this tree.
git diff --name-only $(git merge-base origin/main 699ba570) 699ba570 -> the same 11 owned files, and every one of them is byte-identical to e51ce82b (git diff e51ce82b:<f> 699ba570:<f> empty). Under it: d4d80e98 (#1207), nothing else. The measurement in that review — the tool's own printout carrying all seven record values, the seven JSONs compared to it, the record-mutation bite — is unchanged by a merge, and tools/npu_leaderboard.py still regenerates the doc byte-identically at this head.
The one open nit is the same one: "repeats": 3 on three prompt-window tickets that are one measured window.
PR Reviewer Guide 🔍Here are some key observations to aid the review process:
|
…uard (#1229) #1206's guard compares the seven MTP records' published values against the fixtures, but it names those seven ids in a dict: a record added later -- a new target, a re-measured window, a k=6 row -- would land in bench/npu-leaderboard/ and be ignored by the check, looking exactly like a covered one. That is the "one table row per record" hole #1213 closed for the generated page, one field over, and the same defect this lane keeps paying for: a check that reports success on the records it was told about rather than the records that exist. The guard now also reads the directory: every file whose id starts 27b-mtp- must be one of the ids it derives, and the readable-directory count is asserted first so an empty listing cannot pass as coverage. Measured (pure python, no model, no device): before: the guard reported `all checks passed` with an extra record present on disk. after: an 8th record added as a copy of the hit-rate one -> `FAIL every MTP record on disk is checked by this guard: uncovered ['27b-mtp-extra-20261105']`, `1 CHECK(S) FAILED`; record removed -> `all checks passed`, `the guard's record directory is readable (7 MTP record(s))`. the existing arms are unchanged: 7 of 7 records equal what the fixtures derive (75.14 / 47.51 / 34.81 / 29.28 %, 46.69 %, 2.87 / 2.82), the +0.01 anti-vacuity arm still bites. tools/copyright.py --check rc 0. refs #805 (P6's leaderboard line); extends #1206. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
…test (#1284) Two corrections to the rows merged in #1276, both raised in review: - 'repeats': 7 read as seven repeats of the measurement, but the 7 is the number of steady-state tokens inside ONE request whose median is the recorded value. repeats is now 1 and the instrument says 'one run per arm ... median of its 7 steady-state tokens', which is what was done. (Same mismatch pi-prs flagged on #1206's pooled MTP rows.) - the t2s2 row's notes claimed 'greedy text identical on the three-prompt window'. That window was never run for this pair: what was measured is role-level byte equality with the pb16i path (6/6 roles at layer 0, 8/8 at layer 3, three seeds) plus text equality on the one prompt both arms were asked. The 2026-11-06 re-take that would have closed it was blocked by an unrelated serve holding the engine's device lock, so the notes now name that as the open acceptance step instead of implying it was done. Numbers unchanged: 313 vs 410 ms/token, 1.31x, 1.42x on the device wait, 2.12x on bytes. docs/npu-leaderboard.md regenerated (21 records valid). Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
What was missing
docs/npu-leaderboard.mdis generated frombench/npu-leaderboard/*.json, and it held 12 records with no MTP row: of the P6 gate line "acceptance length, hit rate, passes/token on 8 prompts x 64 tokens per target" and "acceptance numbers per target in docs/npu-leaderboard.md", the numbers existed only as prose indocs/npu.mdand as threenpu-holdticket logs in one home directory. Nothing in the leaderboard — the file the plan names — carried them.What this adds
Seven records, every one derived from the window CI can already read (
tests/fixtures/mtp_clause55, the 8 prompts x 64 tokens, k=4, tau=0 window, 181 chain iterations pooled):Each value is an integer count over the window divided by that window's own denominator — not an average of printed rows. The pool tool reproduces the counts independently (136/181 live at depth 1; 136/181, 84/134, 60/82, 49/59 offline at the loop's own positions), and
settingscarries what makes a cell mean something here:model_dir=bonsai2-27b-pb16i,head=pb16i-exact,k=4,tau=0,batch_rows=16,vocab=248320,hidden=post-final-norm.Two things the records state instead of hiding
Conditioning. The brief's 2 pp clause compares rates conditional on reaching the depth — +0.00 / +0.55 / +0.09 / +1.08 pp, 4 of 4 PASS, which is the verdict already in
docs/npu.md. The leaderboard rows are the all-chains view of the same counts (each depth over all 181 chains), which reads +0.00 / +1.10 / +1.66 / +2.21 pp: depth 4 is where the two conditionings disagree about the bar. Both readings are in the notes; neither is dropped because it is inconvenient.Weighting.
docs/npu.mdsays the chain is "1.80 to 4.33 tokens per pass, mean 3.14". That is the unweighted mean of the eight per-prompt chains; pooled over the window's 181 iterations the same counts give 2.87 (short-chain prompts take most of the iterations). The sentence now names both, so a reader cannot compare 3.14 with a pooled 2.87 and conclude one is wrong.Verified
python3 tools/npu_leaderboard.py --check→ 19 record(s) valid;docs/npu-leaderboard.mdbyte-identical, so the committed file is exactly what the records produce;python3 tests/mtp_depth_pool_test.py→ all checks passed, including the tool's known-answer check, which reads the same fixtures and prints4 of 4 scored depths within 2 pp at the same positions;python3 tools/copyright.py --check→ rc 0.No device was touched for this: it is the recorded window, re-derived. Not in this PR, and worth a follow-up: nothing in CI regenerates
docs/npu-leaderboard.md, so a stale generated file would pass. That is the same class as the fixture hole (#1146) and the compile hole (#1184), and I did not stack it onto this PR because #1184 already changes the same job's steps.