Repository navigation
context: keep a fused op when only its layer's placement disagrees (-fcd) - #194
Conversation
…fcd) resolve_fused_ops disables a fused op for EVERY layer as soon as one of its nodes sits on a device other than its layer's: one mismatching layer in a 41-layer model turns off Flash Attention, the DSV4 hyper-connection trio and the Lightning Indexer for all 41, and the graph is then built unfused. That is the right default (a fused op spanning two devices pays a copy per execution) but there was no way to say "keep it anyway", except for FA, where an explicit -fa on skips the probe by setting auto_fa=false. auto_fhc and auto_flid are hard-coded true in the context ctor and have no flag at all, so the hyper-connection and indexer kernels cannot be reached at partial offload by any command line. This adds one opt-in switch, default off = today's behaviour byte for byte: -fcd / --fused-cross-device (also -no-fcd, LLAMA_ARG_FUSED_CROSS_DEVICE) When set, a placement mismatch is still logged, but the op stays enabled and runs on the backend that claimed it. The field lives at the end of llama_context_params with the other booleans, as that struct asks.
|
Evidence added after opening, on the same head (9c066e0):
|
bong-water-water-bong
left a comment
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY on the code (default path is byte-identical to the pin); the pp/tg table is the lane's, not mine.
Reviewed at 9c066e007.
What I verified myself:
- The default can move nothing. With
fused_cross_device == falsethe new branch is
if (device_mismatch && !cparams.fused_cross_device), which is exactly the oldif (device_mismatch), and the else
branch then runsenabled = truewith the extraLLAMA_LOG_INFOline. So a run without the flag builds the same
graph as the pin; the only difference is one log line per kept op, and only when the flag is on. - The struct change is positionally consistent. I extracted
llama_context_paramsfrominclude/llama.hand the
designated comments fromllama_context_default_params()and compared them in order: 37 fields, 37 initializers, and
every index matches across the insertion. The two apparent mismatches (samplers/sampler,n_samplers/n_sampler)
are the pre-existing alias and are on the pin too. Nothing shifts value-to-field silently. - Compiles:
common/arg.cpp,common/common.cpp,src/llama-context.cppwith the recordedGGML_HRX=ONflags ->
rc 0, 0 warnings each. - The plumbing is complete for every entry point I could find:
common_paramsfield, the-fcd/-no-fcdpair with
LLAMA_ARG_FUSED_CROSS_DEVICE, the mapping incommon_context_params_to_llama,llama_cparams, and the ctor
assignment — which is placed next to the othercparams.*defaults, before anything could overwrite it.
Two notes, neither a gate:
- The body says the field is "placed at the end of the struct with the other booleans". It is not at the end: it goes
afterkv_unifiedand before the[EXPERIMENTAL]sampler block, which is where the booleans live and is the right
place — but this struct is ABI, and adding a field beforesampler/n_samplermoves their offsets. Nothing on this
box links a prebuiltlibllamaagainst a newer header, so it is harmless today; the sentence should not claim
"no field shifts". - Not reproduced by me: the pp/tg table (
-ngl 20 -fcd2135.6 / 65.3 vs 1102.3 / 34.3, three paired repeats) is an
HRX device run and there is no working HRX build on this box after the reboot for me to re-take it. The direction is
consistent with the graph-split counts, which the same log records (113 splits -> 1 at bs256), and with the rule the
code overrides — worth knowing that the numbers rest on that log.
On V4.1 with #190 + #182 stacked under this patch, and what it says about the flagLocal measurement stack (not pushed as a PR — it carries two other lanes' commits): The flag behaves exactly as designed on this model: i.e. the unfused graph's And it is not the cause of the model's remaining blocker. At a fixed budget of 24576 MiB the rule and What that means for this PR: the speed effect measured on Qwen3-0.6B (pp +89 %/tg +39 % at |
`-fcd/--fused-cross-device` (#194) is declared with `set_examples({..., LLAMA_EXAMPLE_BENCH, ...})` in common/arg.cpp, but llama-bench parses its own closed argument set and rejects anything not in its whitelist, so the flag is unreachable from the one tool that measures the placement it changes: $ llama-bench -m model.gguf -fcd -p 512 -n 32 error: invalid parameter for argument: -fcd That leaves the tool unable to reproduce the configuration #194 exists for (partial offload, where the fused op's layer stays on the CPU). Carry the flag the way the neighbouring placement flags are carried: a `cmd_params` field with the same default the library uses (off), the pair of switches, the instance field, `cparams.fused_cross_device` in `to_llama_cparams()`, and a results column that only appears when it is not the default. Verified on gfx1151 (HRX0), Qwen3-0.6B-Q4_K_M, -ngl 99 -p 64 -n 8: before: "error: invalid parameter for argument: -fcd" after : pp64 7497.98 t/s, tg8 361.93 t/s, "fused_cross_device | 1" in the header Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg>
Root cause
llama_context::resolve_fused_ops(src/llama-context.cpp) enables a fused op only if every fused node sits onthe same device as its layer:
One mismatching layer therefore disables the op for all layers. On a model that does not fit the device, that is every
configuration:
-ngl Nleaves layers on the CPU, so Flash Attention, the DSV4 hyper-connection trio and the LightningIndexer are all turned off and the graph is rebuilt unfused — hundreds of extra nodes and a CPU<->device split per
one of them.
The rule is deliberate (a fused op spanning two devices pays a copy per execution), but there was no way to override
it. Flash Attention has an accidental one — an explicit
-fa onsetsauto_fa=false, so the probe never runs — whileauto_fhc/auto_flidare hard-codedtruein the context ctor and have no flag at all. So the hyper-connection andindexer kernels the HRX backend has matchers for cannot be reached at partial offload by any command line.
Fix
One opt-in switch, default off = today's behaviour:
With it set, a placement mismatch is still logged (as
kept enabled across devicesinstead ofset to disabled) but theop stays enabled and runs on the backend that claimed it.
llama_context_params::fused_cross_deviceis placed at the endof the struct with the other booleans, where the struct's own comment asks for them, so no field shifts and the default
is
falseinllama_context_default_params.Files:
include/llama.h,src/llama-cparams.h,src/llama-context.cpp,common/{common.h,common.cpp,arg.cpp}— 6 files,+28/−1.
Measured (Qwen3-0.6B-Q4_K_M, 29 layers, 564-token prompt, one gpu-hold, warm JIT,
-c 1024 -b 256 -ub 256 -t 4 -fit off -n 32)-ngl 99-ngl 4-ngl 4 -fa on-ngl 4 -fcd-ngl 20-ngl 20 -fcdThe four
-ngl 4/-ngl 20rows are medians of three runs per arm. Raw pp/tg per repeat:-ngl 4-ngl 4 -fcd-ngl 20-ngl 20 -fcdEvery one of the three paired repeats agrees in direction on both metrics, so the effect survives the box's noise (within-arm
spread is up to ±25 % pp, which is why medians and pairs are used rather than a single number): pp +89 % / tg +39 % at
-ngl 4, pp +94 % / tg +90 % at-ngl 20. The prefills also collapse from 113 graph splits to 1 at bs=256 in every run.-fcdis-fa onplus the warning. At-ngl 4the two agree to 3.7 % pp / 0.4 % tg (single runs, same session) — theydiffer only in whether
resolve_fused_opsstill runs and logs the mismatch.One caveat kept: the
-ngl 99reference is this build's, not the fastest one. An 11-03 measurement of the same arm on ano-add-on build gave 2726/292, so the add-on's dense-prefill/FA dispatch is slower on this 564-token shape. That is not this
PR's subject; the claim is only the same-build A/B pairs.
-fa onand-fcdare expected to land in the same place: they differ only in whether the warning is still emitted.Perplexity gate on the same 4 chunks of wikitext (keeping an op must move speed, not quality):
-ngl 99-ngl 4-ngl 4 -fcd-ngl 99(repeat)Identical to four decimals, so keeping the fused op moves speed and not quality. Note the direction of the mismatch in this
harness is the other one —
llama-perplexityat-ngl 99reports "layer 25 is assigned to device HRX0 but Flash Attentionis assigned to device CPU", i.e. the node fell to the CPU while its layer is on the device; the flag keeps the fused op
enabled there too and it runs on the CPU, which is why the perplexity is unchanged rather than improved. Both directions are
the same code path.
Why this matters beyond the 0.6B
On the DeepSeek-V4.1-Flash lane the fused ops are claimed by HRX but discarded by this rule at every rung a 124 GiB box
can hold: full offload needs 112.4 GiB of device model buffer and ggml-org#680 measured a device fault at GTT 106.5 of 120 GiB,
so the safe maximum is ~37/41 layers — and 4 CPU layers are enough to disable all of it. With the flag the same hardware
can finally measure what those kernels are worth. Nothing here changes any default, so no other model or backend moves.
Refs (context, not closed by this PR): engine ggml-org#680 (the GTT fault point), ggml-org#788/ggml-org#900/ggml-org#920 (the HRX memory work).