Repository navigation
hrx: make the weight-mapping import opt-in until it stops hanging large loads (engine #1339) - #231
Conversation
…ge loads (engine ggml-org#1339) Fork #226 imports the loader's mapping instead of copying weights to the device (engine ggml-org#856). On a dense 16.4 GiB GGUF and a 76 GiB one the import never finishes: the mapping is fully paged in, then GTT and RSS stay flat and the load waits with no error. GGML_HRX_WEIGHT_HOST_PLACEMENT=1 now opts in; unset or 0 keeps the device copy and the read-only mapping, the placement every control run loaded with. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
ready for review at da22b63 |
bong-water-water-bong
left a comment
There was a problem hiding this comment.
VERDICT: READY at da22b63bac63645eed07b54718ad206793c9c69f. The right mitigation, and the whole of it: one switch now governs both the import and the capability the loader keys its writable mapping on.
An admission first: I READY'd the import's engine pin (engine ggml-org#1303) after testing a 4B copy for file integrity, and took the author's MoE row (the tightness-gate fallback) as the large-model evidence. I never loaded a large dense model with the import on, which is the case that hangs here. This PR is the correct response to that gap.
Checked in the source at this head (ggml/src/ggml-hrx/ggml-hrx.cpp)
import_host_pointer_enabled()(:163) is now true only forGGML_HRX_WEIGHT_HOST_PLACEMENT=1; unset or0means off. A script that set=0for the old A/B control still gets "off".- The device's
props->caps = { true, true, import_host_pointer_enabled(), false }(:866) puts the switch in thebuffer_from_host_ptrslot. So with it off,src/llama.cpp'smap_weights_writable(any non-CPU device withcaps.buffer_from_host_ptr) is false: the loader goes back to the read-only shared mapping and opens no model filer+b. The body's claim holds. device_buffer_from_host_ptr()also refuses early when the switch is off (:915), so even a caller that ignored the capability gets the device copy.- One file, +8/−6. The
static constlambda reads the env once, as before.
Not re-run: the 27B load (11 min stall → 24 s) and the 4B GTT/PPL rows. Those are the author's device rows.
For ggml-org#1339's bookkeeping: this is a mitigation (the import still hangs on large files, now behind =1), and the body says so ("Mitigates"). The engine pin bump ggml-org#1343 says Fixes #1339, which would close the issue while the hang is unresolved. Either keep ggml-org#1339 open with "default-off mitigation landed" or open a follow-up for the hang itself, so the GTT saving isn't quietly lost. ggml-org#1343's pin-on-branch sequencing is already called out there.
…tead of importing the mapping (engine ggml-org#1339) (#234) Reverts the weight-mapping import (#226) and its opt-in switch (#231). The import registered the loader's shared file mapping with KFD as a userptr; KFD always write-faults userptr pages, so the whole file is dirtied, its mtime bumped, and writeback keeps invalidating the registration while it is built. libhsakmt retries the ioctl on EAGAIN, so large models (16.4 GB dense, 76 GB) never finish loading. GGML_HRX_WEIGHT_HOST_PLACEMENT no longer does anything. Instead, when the model uses an HRX device with GPU layers, load_tensors switches the loader to its read() path (same buffer placement as the mapped load) and drops each weight's page-cache pages with posix_fadvise(DONTNEED) once it has been read, so the model is resident once (in the HRX buffers) rather than once in the device buffers and once more in the page cache (engine ggml-org#856). mmap+mlock keeps the mapping. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
Mitigates 1bit-MONSTER/engine#1339 (the engine pin bump carries
Fixes #1339).#226 imports the loader's weight mapping instead of copying weights to the device (engine ggml-org#856). On larger models that import
never finishes. The mapping is fully paged in, then GTT and RSS stay flat and the load waits with no error. This makes the import
opt-in:
GGML_HRX_WEIGHT_HOST_PLACEMENT=1turns it on; unset or0keeps the device copy.caps.buffer_from_host_ptrfollowsthe same switch, so with the import off the loader also goes back to a read-only mapping (
src/llama.cppmap_weights_writable).One file,
ggml/src/ggml-hrx/ggml-hrx.cpp, +8/-6.Measured on strixhalo (gfx1151), one
gpu-holdticket per row group, owndrm-total-gttfrom/proc/<pid>/fdinfollama-cli -ngl 99 -fit off -c 256 -n 8 --temp 0 -no-cnv -p "The capital of France is"; perplexity-f wiki.test.raw -c 512 --chunks 20 -b 512.Before (
a3e8d42c9, shipped default = import on), Qwen3.8-27B-UD-Q4_K_XL (dense, 16.4 GiB), 1 Hz trace:(Flash-Next 76.3 GiB: a 72-minute stall, and a 75 s load with the import off; measured by another lane, engine#856 comment 6707101791.)
After (this branch
da22b63ba):=1(opt-in import)Earlier, on
a3e8d42c9: 4B perplexity with the import on was also 16.3000 +/- 0.82830 (3010 MiB). The import is correct where itcompletes; the problem is that it does not complete on large files.
Cost: the ggml-org#856 GTT saving is off by default again (4B: 3925 vs 1243 MiB) until the import is fixed. Large models load again.