Repository navigation
Update main.cpp - #169
Closed
Anujsharma002 wants to merge 1 commit into
Closed
Update main.cpp#169Anujsharma002 wants to merge 1 commit into
Anujsharma002 wants to merge 1 commit into
Conversation
Deadsg
pushed a commit
to Deadsg/llama.cpp
that referenced
this pull request
Dec 19, 2023
phuongncn
pushed a commit
to phuongncn/llama.cpp-gx10-dgx-sparks-deepseekv4
that referenced
this pull request
Apr 28, 2026
Co-authored-by: Iwan Kawrakow <iwan.kawrakow@gmail.com>
kmbandy
added a commit
to kmbandy/llama.cpp
that referenced
this pull request
Jun 9, 2026
block-seq 256k pd0.01 = 19.6608 wiki / 12.2349 held-out @ 940s — matches the per-target gold on held-out at ~18.5x speed, beats static-single on both. Size curve + speed-multiple climb confirmed; 80k percdamp spread was draw noise. Deferred: clean matched static 256k, big-Hessian sweep, ggml-org#169 lever re-test. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
hjotha
pushed a commit
to hjotha/beellama.cpp
that referenced
this pull request
Oct 25, 2026
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Oct 31, 2026
Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234. Amended r185 (PR ggml-org#169 re-assessment, @briansp2020's finding): bound the aligned depth by a VERIFY WIDTH where there is no expert-cache arena. r153's bound was the arena's band (`band / n_drafting - 1`), and r176 made `band` answer only for a config with host-resident experts -- so on a band-0 config (all-in-VRAM, -sm layer, or a model with no routed expert) a multi-slot draft was UNBOUNDED: four slots at --spec-draft-n-max 7 present ~28-32 rows, the batch leaves the decode kernels' small-batch families, and the DEFAULT recommended speculator (`draft-mtp-adaptive`) runs 9.5-31.8 % slower than the same arm with the depth bounded at 16 rows. The bound is now `verify width / n_drafting - 1`, where the width is the arena band when an arena is active (unchanged) and LLAMA_SPEC_ALIGN_WIDTH_BAND (default 16, 0 = off) otherwise. 16 and not 8: at the two widths the fixed-depth arms measure 16 rows AHEAD on 3 of 4 shapes (35B-A3B 3 GPU 432.7 vs 389.8 t/s, 2 GPU 387.2 vs 359.6, dense 27B 3 GPU 240.1 vs 211.2; qwen4exp Flash-Next GSQ-IQ3_XXS 2 GPU + resident PLE is the exception, 204.6 vs 220.8), so PR ggml-org#169's "above ~8 rows each extra draft token costs more than it gains" does not hold -- the *unbounded* width was the defect, not width 8. 4 concurrent 300-token greedy requests, one binary, band on vs off: 35B-A3B 3 GPU 330.8 -> 434.7 t/s (+31.4 %), qwen4exp IQ3_XXS 2 GPU + PLE 175.5 -> 208.9 (+19.0 %), dense 27B 3 GPU 218.5 -> 249.6 (+14.2 %), and a fixed `draft-mtp` n_max 7 arm 310.9 -> 362.8 (+16.7 %). 3 slots also gains (+16.9 %); 1 and 2 slots are INERT (16 / 2 - 1 = 7 = the default n_max), and the 2-slot text stays byte-identical, so it is a pure multi-slot change. A host-expert config takes the arena band and is untouched (same code path, verified in the log). Gate: scripts/gate-spec-width-band.sh.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 5, 2026
Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234. Amended r185 (PR ggml-org#169 re-assessment, @briansp2020's finding): bound the aligned depth by a VERIFY WIDTH where there is no expert-cache arena. r153's bound was the arena's band (`band / n_drafting - 1`), and r176 made `band` answer only for a config with host-resident experts -- so on a band-0 config (all-in-VRAM, -sm layer, or a model with no routed expert) a multi-slot draft was UNBOUNDED: four slots at --spec-draft-n-max 7 present ~28-32 rows, the batch leaves the decode kernels' small-batch families, and the DEFAULT recommended speculator (`draft-mtp-adaptive`) runs 9.5-31.8 % slower than the same arm with the depth bounded at 16 rows. The bound is now `verify width / n_drafting - 1`, where the width is the arena band when an arena is active (unchanged) and LLAMA_SPEC_ALIGN_WIDTH_BAND (default 16, 0 = off) otherwise. 16 and not 8: at the two widths the fixed-depth arms measure 16 rows AHEAD on 3 of 4 shapes (35B-A3B 3 GPU 432.7 vs 389.8 t/s, 2 GPU 387.2 vs 359.6, dense 27B 3 GPU 240.1 vs 211.2; qwen4exp Flash-Next GSQ-IQ3_XXS 2 GPU + resident PLE is the exception, 204.6 vs 220.8), so PR ggml-org#169's "above ~8 rows each extra draft token costs more than it gains" does not hold -- the *unbounded* width was the defect, not width 8. 4 concurrent 300-token greedy requests, one binary, band on vs off: 35B-A3B 3 GPU 330.8 -> 434.7 t/s (+31.4 %), qwen4exp IQ3_XXS 2 GPU + PLE 175.5 -> 208.9 (+19.0 %), dense 27B 3 GPU 218.5 -> 249.6 (+14.2 %), and a fixed `draft-mtp` n_max 7 arm 310.9 -> 362.8 (+16.7 %). 3 slots also gains (+16.9 %); 1 and 2 slots are INERT (16 / 2 - 1 = 7 = the default n_max), and the 2-slot text stays byte-identical, so it is a pure multi-slot change. A host-expert config takes the arena band and is untouched (same code path, verified in the log). Gate: scripts/gate-spec-width-band.sh.
stew675
pushed a commit
to stew675/llama.cpp
that referenced
this pull request
Nov 7, 2026
Sources: old 01 (whole block). Amended r118 (PR ggml-org#206, @briansp2020): with LLAMA_MTP_INGEST_TAIL set, a prompt ubatch that lies entirely below the ingested tail no longer costs a target readback and a full synchronize. The MTP hook returns before its getters, and llama_get_embeddings_nextn[_ith]() call ctx->synchronize(), so the host used to wait for both backends to drain after every prompt ubatch before it built the next one; the target also copied that ubatch's nextn hidden state to the host (21 MB per 2048-row ubatch on Flash-Next) and waited for it. New staging API llama_set_embeddings_nextn_from(ctx, seq, pos) (default 0 = every row, so other callers are unchanged) marks the rows that are never read: an ingested row p reads h(p - 1), and any ubatch holding a row at pos >= ingest_from - 1 is never skipped. Measured on a 31.5k-token prompt (2x R9700, tail 4096): prefill 1488.6 -> 1509.8 t/s (+1.4 %, 95 % CI +0.9..+1.9), of which the hook half is about +0.6 % and the readback half about +0.1 %; identical greedy text and identical draft counts (177 accepted / 364 generated), decode flat, prefill logits bit-identical, and no change at all with the variable unset. LLAMA_MTP_TAIL_READBACK=0 is the kill-switch (both halves). Amended r125 (issue ggml-org#216): the draft KV cache type now FOLLOWS the main -ctk / -ctv unless the draft type was set explicitly (-ctkd / -ctvd / their env vars), instead of upstream's unconditional f16 default. The draft only proposes (the target verifies), so the greedy text is byte-identical either way, while the draft context's KV shrinks with the main cache: on a dense 27B MTP run at -c 131072 the draft's single attention layer held 512 MiB of f16 KV against 144 MiB at q4_0 -- 368 MiB a 16 GiB card cannot spare. Decode is unchanged within noise -- the quantized draft cache just takes the native quantized-KV FA arm (block 05): three alternating reps per arm gave 56.4-56.6 (f16) vs 56.4-56.5 t/s (q4_0) at -c 8192 -n 400, and 80.8-80.9 vs 80.6-80.7 t/s at -c 102400 -n 3000 (protocol A scale), with ROCBLAS_USE_HIPBLASLT=0. The same-seed greedy text is identical across the two draft KV types and to --spec-type none on all four protocol-A axes (R/P/C/K, -n 3000). Issue ggml-org#216. Amended r153 (issue ggml-org#234, @overdoingism): align the drafts of the sequences that verify in the same step. llama_memory_hybrid::init_batch() splits a decode batch with llama_batch_allocr::split_equal(), which takes one token per sequence set per round and STOPS when the shortest set runs out, so the longer sequence's drafted tail becomes a SECOND ubatch and the target model runs a full extra pass for that one step. The fixed --spec-draft-n-max path never produced a length difference (every slot drafts n_max), which is why this survived, but the adaptive controller holds a PER-SEQUENCE depth: two verifying slots routinely disagree and one decode becomes two passes. draft-mtp-adaptive now gives every drafting sequence the same cap for the call: the deepest cap among them, bounded by the smallest HARD cap in the call (--spec-draft-n-max, the MTP layer cap and the caller's per-slot n_max, which the server lowers as a response runs out of tokens) and by the expert cache's verify band. So no sequence is ever asked for more than it would draft when it runs alone; the only case where a width shrinks is the smallest-hard-cap bound (a response nearing its end), which is transient and keeps the pair in one ubatch. Inert with a single drafting sequence, which is every llama-cli / MTP gate run. Padding the shorter sequence instead is not an option: a padded row advances the recurrent state in mem_recr->prepare(). Amended r154 (issue ggml-org#234): the alignment stays INSIDE the drafter whose caps it changes. r153 also shipped a generic "cut the finished drafts to the shortest" in common_speculative_draft(); measured on the issue reporter's own chain (--spec-type ngram-mod,draft-dflash with a DFlash2 draft, 27B, 2 GPU, recall + prose, -n 200) that is a LOSS: the cut fires on every step whose n-gram match is shorter than the DFlash block, truncates the OTHER slot's good draft, and the pair aggregates 65.6 -> 61.7 t/s (-6 %) while the prose slot's acceptance falls 74 % -> 50 % (splits 12 -> 1, and the lost proposals cost far more than the removed pass). The cut is right only for a mismatch a CAP caused, which align_draft_caps() handles inside the one drafter whose caps those are, so it and common_speculative_impl::draft_truncated() are gone and a chain of different drafters keeps its previous behaviour exactly (measured inert: 64.6 vs 64.4 t/s, identical draft counts). scripts/gate-spec-align.sh grows a --spec-type ngram-mod,draft-dflash arm that fails if the knob is not inert there, so this cannot ship again. Measured (Qwen3.8-27B UD-Q4_K_XL, 2 GPU on the x8 root ports, -sm tensor --parallel 2 --kv-unified -c 32768, greedy, n_predict 400, aggregate t/s of a concurrent high- and low-acceptance pair): 104.8-108.3 -> 140.1-140.6 (+30..34 %), per slot 72-74/57-60 -> 112/76; at 3 GPUs 86.5-87.0 -> 133.1-133.7 (+54 %); on the host-expert shape (35B-A3B UD-Q4_K_M -ncmoe 40) 116.2-116.3 -> 147.7-149.0 (+28 %). The decode steps that split read 58 before and 0 after at every one of those shapes (llama_memory_recurrent's RSPREP trace). The equal-length arm (two identical prompts) is unchanged (-0.4..-2.1 %), a fixed draft-mtp n3 pair measures 143.9 so the fix reaches the equal-length checkpoint within 2 %, and a single drafting sequence is untouched by construction and measured: the n_seq == 1 llama-cli text is byte-identical on/off, as is every single-slot server request. The two-slot pair's low-acceptance slot verifies at its neighbour's width, so the flag's two values are two numeric CONFIGS -- exactly as --spec-draft-n-max 4 and 5 are -- and GREEDY-PURITY.md's promise is greedy text per config, not equality between configs. Measured so that this is not read as noise: every arm reproduces its text character for character across runs (4/4 at 2 GPUs, 2/2 at 3 GPUs and on the host-expert arm), the on/off divergence lands on the same character every time, and a fixed draft-mtp pair at depth 5 against depth 4 moves the same slot the same way with no alignment in play. The token rule is untouched: every emitted token is still the target's own greedy sample at that position. A multi-slot server upgraded across this change can therefore produce different, equally valid text. LLAMA_SPEC_ALIGN=0 is the kill switch. Gate: scripts/gate-spec-align.sh (2 and 3 GPUs, the host-expert arm and a mixed-drafter arm). Issue ggml-org#234. Amended r185 (PR ggml-org#169 re-assessment, @briansp2020's finding): bound the aligned depth by a VERIFY WIDTH where there is no expert-cache arena. r153's bound was the arena's band (`band / n_drafting - 1`), and r176 made `band` answer only for a config with host-resident experts -- so on a band-0 config (all-in-VRAM, -sm layer, or a model with no routed expert) a multi-slot draft was UNBOUNDED: four slots at --spec-draft-n-max 7 present ~28-32 rows, the batch leaves the decode kernels' small-batch families, and the DEFAULT recommended speculator (`draft-mtp-adaptive`) runs 9.5-31.8 % slower than the same arm with the depth bounded at 16 rows. The bound is now `verify width / n_drafting - 1`, where the width is the arena band when an arena is active (unchanged) and LLAMA_SPEC_ALIGN_WIDTH_BAND (default 16, 0 = off) otherwise. 16 and not 8: at the two widths the fixed-depth arms measure 16 rows AHEAD on 3 of 4 shapes (35B-A3B 3 GPU 432.7 vs 389.8 t/s, 2 GPU 387.2 vs 359.6, dense 27B 3 GPU 240.1 vs 211.2; qwen4exp Flash-Next GSQ-IQ3_XXS 2 GPU + resident PLE is the exception, 204.6 vs 220.8), so PR ggml-org#169's "above ~8 rows each extra draft token costs more than it gains" does not hold -- the *unbounded* width was the defect, not width 8. 4 concurrent 300-token greedy requests, one binary, band on vs off: 35B-A3B 3 GPU 330.8 -> 434.7 t/s (+31.4 %), qwen4exp IQ3_XXS 2 GPU + PLE 175.5 -> 208.9 (+19.0 %), dense 27B 3 GPU 218.5 -> 249.6 (+14.2 %), and a fixed `draft-mtp` n_max 7 arm 310.9 -> 362.8 (+16.7 %). 3 slots also gains (+16.9 %); 1 and 2 slots are INERT (16 / 2 - 1 = 7 = the default n_max), and the 2-slot text stays byte-identical, so it is a pure multi-slot change. A host-expert config takes the arena band and is untouched (same code path, verified in the log). Gate: scripts/gate-spec-width-band.sh.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
No description provided.