Skip to content

npu plan P6: the shared drafter protocol - tau-gated chain, 4-row verify with leaf fill, deferred GDN commit; acceptance per target in the leaderboard #805

New issue
New issue
Closed
Closed
npu plan P6: the shared drafter protocol - tau-gated chain, 4-row verify with leaf fill, deferred GDN commit; acceptance per target in the leaderboard#805
Labels
enhancementNew feature or request

Description

@bong-water-water-bong
bong-water-water-bong
opened on Nov 2, 2026

In NpuModel + npu/mtp: chain continues while draft prob >= tau (0.6), verify rows = current + up to 3 drafts, spare rows filled with the drafts 2nd/3rd candidates, GDN commits deferred to the next pass (no per-token state banks). Gate: acceptance length, hit rate, passes/token on 8 prompts x 64 tokens per target; greedy text identical. Public bars: 74 % first-draft, 2.63 tokens/pass (Zhouyi build). Lanes: pi-bonsai then pi-hexagon. Depends on P4.

Part of #799 (the NPU plan layer). Gate and lane per the plan table in #799. Not a c20 blocker (bucket C: sequenced after the release freeze unless marked additive).

Activity

  1. bong-water-water-bong
    added
    enhancementNew feature or request
    on Nov 2, 2026

  2. bong-water-water-bong commented on Nov 2, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 2, 2026
    CollaboratorAuthor

    Acceptance protocol measured against the real base (8 prompts x 64 tokens, k=4, hidden variants compared). This is the
    bar P6 has to keep, and one engine finding that changes what P6 can build.

    Post-final-norm hidden wins; 568 drafted positions:

    depth acceptance
    draft 1 69.4 % (394/568)
    draft 2 58.2 % (328/564)
    draft 3 49.6 % (274/552)
    draft 4 41.4 % (225/544)
    hit rate 0.694
    tokens per pass (unconditional chain) 3.19

    Pre-norm residual hidden: first draft 70.4 % (marginally better at depth 1) but the chain collapses — d2 15.7 %, d3 10 %,
    d4 5 %, i.e. 2.1 tokens/pass. The head feeding itself its own output hidden is only consistent when that hidden is in
    the post-norm basis, so post-norm is the variant; that settles the plan's open item C.

    Versus the published class (74 % first draft, 2.63 tokens/pass): we are a little behind at depth 1 and ahead on tokens
    per pass     (3.19), because our deeper drafts hold up (41 % at depth 4).

    Engine finding — the deferred commit is required, not an optimisation. restore_state() refuses a prefix restore on
    this model: it compares the requested per-position KV length against the conv/rec arrays, which for GDN layers are
    fixed-size windows rather than per-position state, so put() returns false (restore_state(10 of 10) refused). With a
    snapshot-and-replay adapter a live draft-verify loop is therefore either refused (what happens today) or would advance the
    recurrence over rejected rows and silently diverge. So the protocol above is deliberately offline: the head is run over
    the base's own greedy stream with its KV primed position by position — which is how the MTP layer is trained — and each
    draft is compared with the token the model actually produced. That reports the same quantity the published bars report
    without any base-state surgery.

    Conclusion for P6: the loop (#785) and its commit interface (#798, commit_verify(accepted)) are in main, but the engine
    side must defer GDN updates across a verify batch (store the rows' update inputs, apply the accepted prefix) before a
    live speculative loop can be measured. Until then the offline protocol above is the honest number, and it is good enough
    to justify the work.

  3. bong-water-water-bong
    added 11 commits that reference this issue on Nov 4, 2026

    mtp tool: measure the two costs the cost model guessed -- one lm-head…

    721cf90

    mtp: block the head's matvec over n so a code byte shares its cache l…

    f2d9d28

    mtp tool: measure the two costs the cost model guessed -- one lm-head…

    e229e0f

    mtp: block the head's matvec over n so a code byte shares its cache l…

    1120bc0

    tests: the sweep's toy base carries a recurrent state, so a wrong com…

    3bb64cc

    tests: the sweep's toy base carries a recurrent state, so a wrong com…

    7207770

    npu: the draft loop advances over either documented hidden-row shape,…

    6583c69

    npu/mtp: the forward adapter describes the deferral it uses, and its …

    14bda3c

    mtp: the leaderboard records' numbers are re-derived from the fixture…

    e51ce82

    docs: the leaderboard carries the MTP draft loop's measured acceptanc…

    cbf046f

    mtp: a new MTP leaderboard record cannot arrive outside the records g…

    af430d1

  4. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 6, 2026
    CollaboratorAuthor

    Disposition: parked on the phases it depends on — the protocol half is landed and evidenced; the remaining rows are not measurable yet.

    On origin/main (19486dcc): the P6 protocol lives in npu/mtp/draft.h (tau=0.6 default, reached_at vs accepted_at, deferred GDN commit) and is exercised by tests/mtp_draft_test.cpp; the leaderboard carries the live-loop acceptance rows from #1206 — depth1 75.14 %, hit rate 46.69 %, 2.87 tokens/pass, 8 prompts x 64 tokens, k=4, tau=0, greedy text 8/8 identical. (PR #1229, the leaderboard records guard, is merged and is not this protocol's PR.)

    Remains: the tau=0.6 row, the Hexagon rows (P5 is parked: #804, blocked on private kernels/hardware), and the NpuModel side of the protocol, which depends on P4 (#803, PR #808, waiting on the release freeze).

    Unpark: P4 lands so the protocol has its model-side target, and the P5 hardware path exists; the tau=0.6 and Hexagon rows then become measurable and this phase's acceptance is complete.

  5. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 6, 2026
    CollaboratorAuthor

    Scoped PR for this plan: #1277 (737dad46, refs #805) — the measurement window now carries tau.

    Why that is the shape of the remaining work. Options::tau defaults to 0.6, but every published acceptance row is tau = 0: the live path printed no window identity and tools/mtp_depth_pool.py never read tau, so a tau=0.6 ticket could be pooled with a tau=0 one and the deeper rows (offered only by tau=0 chains) would read as a complete pass. #1277 prints == N prompts x M tokens, k=K tau=T, hidden=H at init on the live path too, has the tool name a mixed pool and report tau unstated for older logs, and keys the leaderboard-records guard by window.

    What remains on P6, and why each row is blocked rather than skipped

    row state
    tau-gated chain landed (default 0.6, npu/mtp/draft.h), exercised by tests/mtp_draft_test.cpp
    4-row verify, leaf fill not built. The verify batch is flat: row m is position start_pos+m and is scored against row m-1's logits, so a draft's 2nd/3rd candidates cannot be placed at the same position without per-row position ids and an attention mask in the forward — a kernel-side change (private repos), and it needs P4's NpuModel to target.
    deferred GDN commit landed (restore_state() refuses a prefix restore; the verify kernel stores per-token update inputs)
    acceptance / hit rate / tokens per pass, 8×64 landed at tau = 0 (#1206: depth 1 75.14 %, hit 46.69 %, 2.87 tokens/pass, text 8/8 identical)
    the same at the default tau = 0.6 unmeasured. Needs one 8×64 --live --match --tau 0.6 window (~4 npu-hold tickets of ≤10 min). Blocked right now: /tmp/1bit-npu-device.lock is held by an orphaned host-GEMM serve (pid 2231177, ~/lb/1035-row) that pi-bonsai has asked its owner to stop, and c31's gate chain is running (TISSUES-ALL rule 1). My 15:17 probe ticket refused with device lock busy rather than running beside it — which is the harness working as designed.
    Hexagon rows (P5) blocked on #804 (private kernels/hardware)
    NpuModel side of the protocol blocked on P4 (#803, PR #808)

    Nothing here is closed by hand; #1277 stays refs rather than Fixes because the tau=0.6 and Hexagon rows are still open.

  6. bong-water-water-bong
    added a commit that references this issue on Nov 6, 2026

    mtp: the measurement window carries tau, so two chain gates cannot be…

    a4447ee

  7. bong-water-water-bong commented on Nov 6, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 6, 2026
    CollaboratorAuthor

    The tau=0.6 row is measured and published: PR #1300 (on 5c496d32 → b0224e5d), 8 prompts x 64 tokens, k=4, one prompt per npu-hold ticket, text identical 8/8, 0 stream mismatches, amdxdna 0 in every arm.

    tau = 0 tau = 0.6 (the default)
    hit rate 46.69 % (338/724) 78.77 % (230/292)
    depth 1 acceptance (all chains) 75.14 % (136/181) 78.34 % (217/277)
    depth 2 47.51 % (86/181) 3.97 % (11/277)
    depth 3 34.81 % (63/181) 0.72 % (2/277)
    depth 4 29.28 % (53/181) never offered
    tokens per pass 2.87 1.83

    The gate buys confidence per draft and pays for it with the chain: ~1.05 drafts per chain against 4.0, so 1.83 tokens/pass sits below the measured 3.2-4.1 break-even and the default is a 0.81x slowdown. Depth 1 is the only depth with a usable denominator at this tau (depth 2 has 13 chains, depth 3 has 2, depth 4 none).

    So the gate rows of this issue are now closed at both taus. What remains is unchanged and honestly blocked: leaf fill (the verify batch is flat — row m is start_pos+m and is scored against row m-1's logits, so a draft's 2nd/3rd candidates need per-row positions and an attention mask in the forward, on top of P4/#803), and the Hexagon rows (P5, #804).

  8. bong-water-water-bong
    added 2 commits that reference this issue on Nov 6, 2026

    copyright: a fixture ticket under tests/fixtures/ is exempt by KIND, …

    6fc93fa

    mtp: the draft route's DEFAULT tau=0.6 is measured — 78.8 % hit rate …

    b7e3c37

  9. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 7, 2026
    CollaboratorAuthor

    P6 coordination: what P4 gives the drafter today, and the one row that still waits on A3

    P4/A1 merged as 0a2cbb97 (npu/target/target.h + npu/target/xdna/ + q4nx_lh, the launch layer behind NpuTarget). For P6 that means, precisely:

    • Available now: the launch layer is no longer inside Q4nxNpuForward, so a draft/verify pass reaches the device through NpuTarget::gemv(GemvArgs{role, layer, M, K, N, x, Bmat, wkey, tag, rows, decode_twin}) — note rows and decode_twin are already in that struct, which is the pair a 4-row verify pass needs (the M=16 twin streams its weights once for the batch). npu/mtp/draft.h is untouched by the move, as the interface was designed to leave it.
    • Not available yet: the plan row that reads "deferred GDN commit in NpuModel" still waits on A3 (npu plan P4: extract NpuTarget + NpuModel; xdna target wraps I8Ctx; host fallbacks for optional ops (lands after c20 ships) #803, reopened): NpuModel — the model maths behind NpuTarget& with the forward as an adapter — is not in the tree. Today the maths still lives in Q4nxNpuForward, so the deferred-commit ownership has not moved and a P6 row that asserts it would be asserting A1.
    • Also waiting on P5: the per-target acceptance row. --npu-target <name> exists end to end (compiled list in app/forward_serve.cpp, refusal by name before the device opens), so the Hexagon arm of the acceptance table can be selected by that flag the moment pi-hexagon's target links in — no new plumbing on your side.

    Your tau=0.6 row (#1300: 78.77 % hit rate against 46.69 % at tau=0, depth-1 acceptance 78.34 %, 1.83 tokens/pass) is the number the umbrella needed for P6 and is already reflected in my #799 checklist. What I would keep out of P6 until A3 lands: any row whose claim is about where the deferred commit lives, as opposed to what the acceptance is.

  10. bong-water-water-bong
    added a commit that references this issue on Nov 7, 2026

    mtp: leaf fill cannot buy acceptance in this verifier — P6's last pla…

    36fb301

  11. bong-water-water-bong commented on Nov 7, 2026

    @bong-water-water-bong
    bong-water-water-bong
    on Nov 7, 2026
    CollaboratorAuthor

    P6 is complete — closing. Every part of this issue's protocol is landed and measured, and the one item never built (leaf fill) is now rejected on evidence, not implemented: a test in tests/mtp_draft_test.cpp [16] shows it cannot buy acceptance in this verifier. All evidence below is re-checked at origin/main 0f45221e, not quoted from memory.

    The protocol, part by part

    part state evidence
    tau-gated chain (default 0.6) landed npu/mtp/draft.h (Options::tau, reached_at vs accepted_at), tests/mtp_draft_test.cpp [6]/[9]
    deferred GDN commit, no per-token state banks landed restore_state() refuses a prefix restore; the verify batch stores per-token update inputs and commit_verify(accepted) applies only the accepted prefix
    4-row verify, greedy text identical landed flat batch step_batch_all, one M=k+1 batch per iteration; text identical to plain decode on 8/8 prompts at both gates
    acceptance / hit rate / tokens-per-pass, 8 prompts × 64, per target landed 13 27b-mtp-* leaderboard records, every one re-derived from in-tree fixtures by tests/mtp_depth_pool_test.py
    leaf fill ("spare rows with 2nd/3rd candidates") rejected with a test see below

    Measured (8 prompts × 64 tokens, k=4, one prompt per npu-hold ticket, text identical 8/8, 0 stream mismatches, amdxdna 0 in every arm)

    tau = 0 tau = 0.6 (the default)
    hit rate 46.69 % (338/724) 78.77 % (230/292)
    depth 1 acceptance (all chains) 75.14 % (136/181) 78.34 % (217/277)
    depth 2 / 3 / 4 47.51 / 34.81 / 29.28 % 3.97 / 0.72 % / never offered
    tokens per pass (live) 2.87 1.83
    depth-1 live vs offline, same positions +0.00 pp PASS +0.00 pp PASS

    Reproduce: python3 tools/mtp_depth_pool.py tests/fixtures/mtp_clause55/*.txt and … tests/fixtures/mtp_tau06/*.txt (each prints the all-chains rates, hit rate and both tokens-per-pass figures; --selftest checks the recorded counts).

    The finding worth keeping: at its default the gate raises confidence per draft (46.7 % → 78.8 % hit rate) and collapses the chain (~1.05 drafts per chain against 4.0), so 1.83 tokens/pass sits below the 3.2–4.1 break-even and the draft route is a 0.81× slowdown as configured. Serve it at tau = 0 or not at all; docs/npu.md says so.

    Leaf fill — why it is not built

    Acceptance is argmax(logits[i-1]) == draft[i] (npu/mtp/draft.h:317): a draft is accepted iff it is the base's own argmax, and the argmax grading depth i comes from row i-1 of the same batch. Measured in [16] against a toy base with a real two-level ranking:

    rank-2 candidate at every depth ................. accepted 0/92
same head, argmax moved onto the runner-up ...... accepted 20/20     (control: the zero is the rule, not the harness)
right at depths 2..k, wrong at depth 1 .......... accepted 0, reached_at[1] == 0
the same head at k=8 (M=16 rows) ................ accepted 0 of 184 rows

    So a non-argmax candidate at the same position is refused with certainty, a correct candidate under a refused parent is never scored, and widening the batch buys nothing — the rows are the chain, not candidate tests. A real tree verifier would need per-row positions + an attention mask in the forward, and with a row-by-row GDN state a state per branch: the "no per-token state banks" constraint this issue itself imposes. If that constraint is ever lifted, leaf fill becomes a new design and should get its own issue.

    Merged in main for this issue

    #820 (the drafter's stream cursor), #838 (the live-vs-offline comparison), #968, #1022 (the replay is 16× cheaper), #1042, #1206 (the leaderboard rows), #1213 (the page regeneration guard), #1229 (the records guard covers the directory), #1277 (the window carries tau), #1300 (the default tau=0.6 measured), #1305 (this determination).

    Not in this issue: the Hexagon per-target rows belong to P5/#804, and the NpuModel side landed with P4 (#808) (0a2cbb97).

  12. bong-water-water-bong
    closed this as completedon Nov 7, 2026
  13. bong-water-water-bong
    added 4 commits that reference this issue on Nov 7, 2026

    mtp: a depth too thin for the 2 pp bar says so, instead of printing a…

    fa44855

    mtp: the draft route at main — acceptance re-confirmed digit for digi…

    f46562f

    npu docs: the round's cost, split — base passes are 69-82 % of it, an…

    5e742d3

    npu docs: the drafter's break-even quoted with its loop — 1.17 tokens…

    e0ca7d6
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions