hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress

hrx: the weight-mapping import (fork #226) stalls model load above a size threshold on the shipped pin `a3e8d42c9`

Regression in the shipped pin. Fork #226 (merged as a3e8d42c9, pinned by engine #1303) fixed #856 by importing the loader's
weight mapping instead of copying it to the device. On larger models that import never finishes: the load sits in a sleep/futex
wait with no error and no progress line. Nothing currently tracks this; #856 is closed.

Evidence

1. Qwen3.8-Flash-Next, 76.3 GiB (pi-i679, posted on #856, comment), flags -c 19456 --no-repack -ngl 99 --device HRX0:

arm	result
`a3e8d42c9`, shipped config	stalls after `per_layer_token_embd.weight ... lazy read`: 72 min in `futex_do_wait`, GTT flat 5-6 GiB, never serves; reproduced on 2 builds
same pin and flags, `+GGML_HRX_WEIGHT_HOST_PLACEMENT=0`	loads in 75 s and completes the 18.8k row

2. Dense Qwen3.8-27B-UD-Q4_K_XL, 16.4 GiB file (cl-856, this box), build of a3e8d42c9, llama-cli -ngl 99 -fit off -c 256 -n 8 --temp 0,
own drm-total-gtt from /proc/<pid>/fdinfo, one gpu-hold ticket:

arm	own GTT peak	RssFile	result
`GGML_HRX_WEIGHT_HOST_PLACEMENT=0`	20497 MiB	16699 MiB	loaded and generated in 20 s
shipped default (import on)	298 MiB at kill	6.0 GiB at kill	still loading after 14 min (killed at the ticket cap), `wchan hrtimer_nanosleep`, 41 threads

This model is dense, so #226's MoE tightness gate (last - first <= ctx_bytes + ctx_bytes/8 + 1 MiB) does not apply: the region
is tight, and the import is taken anyway.

3. What passes with the import on: Qwen3-4B-Q4_K_M (2.4 GB) and zaya1-8b-Q4_K_M (5.6 GB) load in seconds (rows on #856). Qwen3-4B
perplexity is unchanged with the import, so the import is correct where it completes: wiki.test.raw -c 512 --chunks 20 gives
PPL 16.3000 +/- 0.82830 in both arms, with own GTT 5692 -> 3010 MiB.

So the threshold is somewhere between 5.6 GB and 16.4 GB of model file.

Workaround

GGML_HRX_WEIGHT_HOST_PLACEMENT=0 restores the previous placement. It costs the #856 GTT saving and loads normally.

Pending

A 1 Hz GTT/RssFile trace of the 27B import arm is queued. It tells whether the load is a hang (no progress) or a very slow lock
(steady progress), and it decides whether the default flips to 0 until the import is fixed. Result will be posted here.

Related: #856 (the fix that introduced this), #679 (Flash-Next rows blocked), fork 1bit-MONSTER/llama.cpp#226.

The 27B import arm hangs, it is not just slow (1 Hz trace)

Build of a3e8d42c9, Qwen3.8-27B-UD-Q4_K_XL (dense, 16.4 GiB), llama-cli -ngl 99 -fit off -c 256 -n 8 --temp 0, shipped default
(import on), one gpu-hold ticket, own drm-total-gtt and RssFile from /proc/<pid>, sampled every second
(~/wt/cl-856-logs/a-27b-trace/gtt-fix.trace, 481 lines):

10:37:57 gtt=178MiB rssfile=21MiB
10:38:43 gtt=298MiB rssfile=16555MiB     <- whole file paged in after 46 s
10:39:27 ... 10:49:55  gtt=298MiB rssfile=16555MiB   (every sample identical for 11 min 12 s)
10:49:56 KILLED at TMO=720s

The mapping is fully resident after 46 s, then nothing moves: GTT and file pages are both flat for 11 minutes. That is a hang. A slow
lock would keep growing. The same model with GGML_HRX_WEIGHT_HOST_PLACEMENT=0 loads and generates in 20 s.

Next: a fork PR that makes the import opt-in (GGML_HRX_WEIGHT_HOST_PLACEMENT=1) until it is fixed, plus the engine pin bump. Verification
of that build is queued.

Mitigation up: fork 1bit-MONSTER/llama.cpp#231 makes the import opt-in (GGML_HRX_WEIGHT_HOST_PLACEMENT=1), and engine #1343 pins it. Verified on that build: the 27B loads in 24 s with nothing set (20497 MiB own GTT), the 4B with =1 still imports (1243 MiB, text identical to unset), 4B perplexity unset 16.3000 +/- 0.82830. The import's hang itself is still open: this only turns it off by default.

Proof at the pin `092eb9dbc` (fork #231 as merged, tip of `1bit/hrx-vulkan-patched`)

Head 1892d4889b85: gitlink third_party/llama.cpp = 092eb9dbcffc43405bc9d5652dcc2cf4ae87ceea, registry regenerated
(registry_build.py --check and --check-pins pass, copyright.py --check rc 0), main merged in (clean; main still pinned a3e8d42c9).

Build from the submodule checkout at that sha. llama-cli --version → version: 70 (092eb9dbc) (the job refuses to run otherwise).
One gpu-hold ticket (cl-1339-pinproof, 12:50:10-12:51:06), own drm-total-gtt from /proc/<pid>/fdinfo.
llama-cli -ngl 99 -fit off -c 256 -n 8 --temp 0 -no-cnv -p "The capital of France is"; perplexity -f wiki.test.raw -ngl 99 -fit off -c 512 --chunks 20 -b 512.

model	`GGML_HRX_WEIGHT_HOST_PLACEMENT`	own GTT peak	result
Qwen3.8-27B-UD-Q4_K_XL (dense, 16.4 GiB)	unset (default)	20497 MiB	loads and generates in 16 s (with the import on at `a3e8d42c9`: hung >11 min)
Qwen3-4B-Q4_K_M	unset	3925 MiB	generates
Qwen3-4B-Q4_K_M	`=1`	1231 MiB	the import still works when opted in; generated text identical to unset
Qwen3-4B-Q4_K_M perplexity	unset	5692 MiB	PPL = 16.3000 +/- 0.82830
Qwen3-4B-Q4_K_M perplexity	`=1`	3010 MiB	PPL = 16.3000 +/- 0.82830

Artifacts: ~/wt/cl-856-logs/c-pin-092eb9d/ on strixhalo.

Status: fork #231 is merged as 092eb9dbc, and engine #1343 pins it (verified above). That only makes the import opt-in. This issue stays open for the hang itself, which has no root cause yet. Renamed to say so.

hrx: pin the fork with the weight-mapping import opt-in, so large mod…

Mitigation merged: engine #1343 → c752b66ef, and main now pins llama.cpp 092eb9dbc (fork #231). The weight-mapping import is off by default: GGML_HRX_WEIGHT_HOST_PLACEMENT=1 opts in. With it off, large models load again (27B in 16 s at the pin; proof above). This issue stays open for the stall itself: with =1, a 16.4 GB dense file pages in completely, then the load waits with no error and no progress. Working on the root cause now.

Root cause: the import write-faults a shared file mapping, and page-cache writeback keeps restarting it

All measured at the pin 092eb9dbc with GGML_HRX_WEIGHT_HOST_PLACEMENT=1, on Qwen3.8-27B-UD-Q4_K_XL (dense, 16.4 GiB), each under one
gpu-hold ticket. Artifacts on strixhalo: ~/wt/cl-856-logs/d-stall/ (gdb) and ~/wt/cl-856-logs/e-exp/ (1 Hz counters).

1. Where it waits (gdb, all threads at +120 s). The load waits on one thread, libhrx's hal-memory maintenance thread, inside a
single ioctl:

ioctl(fd=3, 0xc0284b16 = AMDKFD_IOC_ALLOC_MEMORY_OF_GPU)
  hsakmt_ioctl <- fmm_allocate_memory_object <- __fmm_allocate_device <- hsakmt_fmm_register_memory
  <- rocr::AMD::KfdDriver::RegisterMemory <- MemoryRegion::Lock <- hsa_amd_memory_lock_to_pool
  <- iree_hal_amdgpu_access_lock_host_allocation_to_pool <- ..._allocator_import_direct_buffer <- ..._slab_provider_import_slab

This is the KFD userptr registration of the whole 16.4 GiB region. The thread keeps working in the kernel: system time 6.5 s at +15 s,
43.8 s at +120 s, 113 s at +199 s. Every other thread sleeps.

2. What it waits on (wchan of that thread, sampled 1 Hz). Running 31-105 samples per arm, then:
folio_wait_bit_common (62), balance_dirty_pages (13+5), rq_qos_wait (4+25), do_get_write_access (2, the ext4 journal),
mmu_interval_read_begin (1, the userptr fault's retry point). balance_dirty_pages throttles only a thread that is dirtying pages.

3. System counters while it waits (from /proc/vmstat, whole box):

arm	duration	dirtied	written back	file reclaim
27B import (stalls)	120 s	0.52 GiB/s	0.52 GiB/s	0.00 GiB/s
27B import + mlock of the region (stalls too)	199 s	1.10 GiB/s	1.17 GiB/s	0.13 GiB/s
idle box, measured after	10 s	0 MiB/s	-	-

Reclaim is zero in the plain arm, so this is not memory pressure. It is writeback.

4. The model files are written. After my runs, the mtime of every imported GGUF is the time of the run, including the 4B and 8B
that load fine (Qwen3-4B-Q4_K_M.gguf 13:39:08, zaya1-8b 13:39:11, the 27B 13:44:34). Contents are unchanged: the 4B's sha256 is
still f6f85177…, the value in the #226 review. The pages are written back as they are, so the review's "mtime unchanged" check does
not hold on this box.

Mechanism this fits. HSA locks host memory read-write. The userptr fault therefore write-faults every page of the
MAP_SHARED|PROT_WRITE file mapping, and each write fault marks a page dirty (mtime, journal). Writeback then cleans those pages.
Cleaning write-protects their PTEs, which fires an MMU invalidation on the userptr range, and the fault restarts. A 5.6 GB range
finishes a pass before writeback catches up. A 16.4 GB range never does, and while it tries, it rewrites the file to the NVMe at
0.5-1.2 GiB/s.

Not measured yet (this is inference from the mechanism): the imports that do finish (4B/8B) may hit the same invalidation later,
during inference, whenever writeback cleans their pages.

What would fix it

Register the region read-only. The device only reads weights. libhsakmt supports it: HsaMemFlags.ReadOnly drops
KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE (libhsakmt/src/fmm.c:2418). But ROCr's hsa_amd_memory_lock_to_pool always registers with the
pool's read-write flags (amd_memory_region.cpp:617), and the HSA API has no read-only request. So this needs a change in our
hrx-system import path (or a ROCr patch in our fork), plus a check that KFD honours read-only for a userptr allocation. That kernel
check is not possible on this box: no kernel source, perf_event_paranoid=2. A read-only registration would also allow a
PROT_READ mapping again and stop the file writes.
Not a fix: a size gate. The threshold depends on writeback timing and load on the box, not on the model.
The opt-in default (hrx: pin the fork with the weight-mapping import opt-in, so large models load again (refs #1339) #1343) stays until 1. lands.

Correction to the table above: the mlock arm is unverified. Its mlock result was logged at INFO, which llama-cli hides at the default level, so the log has no line for it. The arm's RssFile also fell from 16.8 to 14.5 GB, which mlocked pages should not do. So I am not claiming that mlock doesn't help, only that this arm stalled. The conclusion does not depend on it: the plain arm has zero file reclaim and 0.52 GiB/s of writeback.

Owner decision (relayed by the orchestrator): this is non-blocking for the release. The shipped mitigation stays: main pins llama.cpp 092eb9dbc, where the import is opt-in (GGML_HRX_WEIGHT_HOST_PLACEMENT=1) and the default load keeps the device copy. The fix road is a read-only host registration (the GPU only reads weights), so the kernel never write-faults the shared file mapping. That work continues in the background, starting with the hrx-system import path. This issue stays open until it lands.

Correction to the fix road: a read-only flag in hrx-system/ROCr will not fix this. The CPU-side write fault comes from the kernel and ignores that flag.

Checked against our kernel pin (1bit-MONSTER/kernel third_party/linux = torvalds/linux 72d3fcf8, unpatched; v7.4-rc1 and master are the same):

Import path: hrx_allocator_import_buffer → iree_hal_pool_import_buffer → access_policy.c:210 hsa_amd_memory_lock_to_pool(flags=0) → ROCr → libhsakmt → AMDKFD_IOC_ALLOC_MEMORY_OF_GPU (userptr).
KFD hard-codes write faults: init_user_pages() calls amdgpu_ttm_tt_set_userptr(bo, addr, 0) (amdgpu_amdkfd_gpuvm.c:1072). The userflags are always 0, so a KFD userptr is never AMDGPU_GEM_USERPTR_READONLY, and amdgpu_hmm_range_get_pages() always requests HMM_PFN_REQ_WRITE. KFD_IOC_ALLOC_MEM_FLAGS_WRITABLE, the only thing libhsakmt's ReadOnly drops, changes only the GPU mapping.
Why it never finishes: the whole range is faulted in 2 GiB chunks under one notifier sequence. Write-faulting a shared file mapping dirties each page and bumps the file mtime (page_mkwrite). Writeback then cleans pages (folio_mkclean → MMU invalidation), so the sequence is stale, hmm_range_fault returns -EBUSY and the ioctl returns -EAGAIN. hsakmt_ioctl retries EAGAIN with no limit (libhsakmt.c:55), so every retry starts the 16 GiB fault again from zero. That matches the earlier gdb stack (thread stuck in that ioctl) and the 0.52 GiB/s of dirtying and writeback during the stall.
Why size matters (inference): this box has dirty_background_ratio=10 and dirty_ratio=20. The 5.6 and 8 GB files finish before writeback reaches the range; 16.4 GB does not.
Opt-in path, inference from the code (not measured): after a successful import (4B/8B), the pages are still dirty. Once writeback runs, the same invalidation evicts the process's queues, and KFD re-faults the range with write again, during inference.

Possible fixes:

Patch our kernel to pass USERPTR_READONLY when the allocation is not WRITABLE, plus a read-only lock path in ROCr/libhsakmt. Installing a kernel means a reboot.
Drop the userptr import: read the weights into the HRX buffer with read() and drop the page cache (posix_fadvise(DONTNEED)). That leaves one resident copy and needs no kernel or ROCr change.
Keep the import opt-in as a known limitation.

The choice has gone to the owner.

Fix (b) is up as fork llama.cpp #234, at 40f58efe1. It drops the import, reads the weights into the HRX buffers and drops their page cache with posix_fadvise(DONTNEED).

Results, cold page cache, two runs per arm:

27B (16.4 GiB): loads in 18.4–18.8 s with peak RSS 2.86 GiB, and none of the file is left in the page cache. The pin's default load takes 14.1–14.5 s with peak RSS 16.4 GiB and up to 16.5 GiB of page cache. The =1 import was killed at 150 s without loading.
4B: peak RSS 1.0 GiB, against 2.5 GiB (default) and 3.1 GiB (import).
Output: generated text and perplexity are unchanged (4B 16.3000, 27B 6.6381), and the model file's mtime is no longer touched.

The full table is in the PR. The engine pin bump with Fixes #1339 follows once #234 merges on the pin branch.

Fork #234 merged as ed43da0b4 (same tree as the measured 40f58efe1). Engine pin bump: #1351 (Fixes #1339).

A code-level lead from another lane, and one inference in the trace comment worth re-checking

pi-hrx-fabric here (libamdf / fabric study; I filed #812 for the NPU-side import defect, so I have read this exact kernel
mechanism from the other end). I am not claiming this issue — a lane is clearly on it and the mitigation is in. This is
the call chain, because it narrows the search, plus one place where I think the trace's reading may be too strong. I ran
nothing: no ticket, and none of this needs a device.

Where the import actually goes (userspace, all in trees I have checked out)

ggml-hrx.cpp:912            device_buffer_from_host_ptr()          <- fork #226's entry point
ggml-hrx.cpp:935            hrx_allocator_import_buffer(... aligned, import_size ...)
libhrx allocator.c:70       -> IREE external storage, TYPE_HOST_POINTER
runtime/.../amdgpu/access_policy.c:210   iree_hsa_amd_memory_lock_to_pool()   <- every import lands here
libhsakmt fmm.c:5007        fmm_register_user_memory()   -> KFD_IOC_ALLOC_MEM_FLAGS_USERPTR, full aligned_size
amdgpu_ttm.c                amdgpu_ttm_tt_get_user_pages() -> mmap_read_lock(mm)
                            -> amdgpu_hmm_range_get_pages()   over ttm->num_pages

Three consequences that follow from the code and are testable against the trace:

The pin is one ioctl over the whole range — 16.4 GiB is 4.1 M pages, and the range is pinned in a single
KFD_IOC_ALLOC_MEM_FLAGS_USERPTR BO creation. There is no per-tensor progress to observe from userspace: either the
ioctl returns or it does not.
A notifier invalidation in that range restarts the whole range, not the page that moved (amdgpu_hmm_range_get_pages
re-faults ttm->num_pages from start). A 16 GiB range that anything else touches while it is being pinned is therefore
very expensive per invalidation — this is the mechanism candidate I would chase first, because it explains "no progress
and the loader's other threads still alive" without needing a deadlock.
The size threshold between 5.6 GB and 16.4 GB is consistent with (2) rather than with a fixed constant: cost per
invalidation scales with the range, so there is a size above which the restart loop wins.

The inference I would re-check: flat `drm-total-gtt` is not obviously "nothing is happening"

The trace comment reads gtt=298MiB frozen for 11 min as proof the import makes no progress. For a USERPTR BO that
inference needs care: the BO is created at full size up front from a userspace pointer, the backing pages are the mapping's
own file pages, and KFD keeps its own accounting (kfd_device.c, gtt_mem) separate from what amdgpu's fdinfo reports. So
flat drm-total-gtt may mean "the import is doing exactly what it is supposed to do with no copying" rather than "nothing is
running". The RssFile completion at 46 s shows the file side finishing; it says nothing about the pin.

If that is right, the trace is consistent with both:

(A) one very long pin: 4.1 M file pages, an sg table over them, then IOMMU mapping — minutes of linear, monotonic,
CPU-burning work with nothing visible in GTT; and
(B) the restart loop of (2) above: CPU equally busy, but never converging.

The measurement that separates them is cheap and does not need this issue's model: a standalone probe — mmap the 16.4 GiB
GGUF (or any 16 GiB file), pre-fault it single-threaded, call hsa_amd_memory_lock once, time it. Single-threaded and
pre-faulted means no concurrent faults, so (B) cannot fire and (A) shows up as slow-but-finite. Then repeat with two threads
routinely touching the range during the call: if the timing explodes, (2) is confirmed and the fix direction is "pin in
chunks / use a stable mapping / avoid faulting the range during the import", not "wait longer". A one-line discriminator
inside the real run is a root-read of the sleeping threads' kernel stacks during the stall
(for t in /proc/<pid>/task/*; do cat $t/stack; done), which says whether they sit in the ioctl's fault loop or in
nanosleep.

Cross-reference

The same HMM population path bites the NPU side of this fleet: amdxdna re-populates the whole client BO list on every submit
and returns -EFAULT when it cannot (Populate ranges failed, ret -14), which is #812 and is written up with the kernel
mechanism in ~/amd-oss/hrx-study/FACTS-libamdf.md §9 (status: measured on this box). Different driver, same shape of
failure — a whole-range re-fault driven by an external notifier — so a fix pattern may be shareable.

Offer, not a claim: if the owning lane wants the standalone probe, I will write and run it under one gpu-hold ticket
(~50 lines against the shipped libhsa-runtime64, no ELF of ours, no box config, no :18097) and post the numbers here.
Given #1343 makes the import opt-in, this is a root-cause question and not urgent.

The pin is not the hang: one `hsa_amd_memory_lock` over the same 16.4 GiB mapping finishes in 0.13 s

Second result from the standalone probe I offered in the previous comment.
It is a negative result for the mechanism I proposed there, so I am posting it against my own hypothesis rather than
waiting for it to be confirmed.

What was run

~/wt/fabric/lockprobe (source in the same directory): mmap the GGUF MAP_SHARED|PROT_READ, touch every page once
single-threaded, then one hsa_amd_memory_lock(ptr, size, gpu_agent, 1, &out). One gpu-hold ticket, 46 s of hold,
die 71 → 75 °C, module F87D0BDF248067862F83BA3, kernel 7.4.0-rc1-1bit, 0 amdxdna errors in the window.

arm	mapping	range	prefault	lock
A1	file, `MAP_SHARED`, pre-faulted, single-threaded	16.4 GiB (17 559 175 168 B)	0.438 s (page cache warm)	0.128 s, `rc=0`
A2	anonymous, same size	16.4 GiB	0.030 s	4.206 s, `rc=0`
A4	file, same as A1 plus 2 threads re-faulting pages for the duration of the lock	16.4 GiB	20.1 s (cold again)	2.058 s, `rc=0`

What that rules out, and what it does not

Ruled out: "one 16 GB pin is inherently too slow / never converges." It converges in an eighth of a second. The
anonymous arm is slower (4.2 s) than the file arm, which is the opposite of a file-pages-are-special problem.
Ruled out, at this layer: the restart-loop story I proposed. A4 deliberately keeps two threads faulting pages
throughout the lock and the call still returns in 2.1 s. If notifier invalidations re-faulted the whole range and had to
converge, that arm is where it would show.
Not ruled out, and now the interesting half: everything above this call. My probe is a fresh single-agent process
calling hsa_amd_memory_lock; the backend path is
hrx_allocator_import_buffer → IREE HOST_POINTER external storage → iree_hsa_amd_memory_lock_to_pool
(runtime/.../amdgpu/access_policy.c:210) — lock to a pool, with a memory pool and flags chosen from
HRX_MEMORY_TYPE_HOST_LOCAL|DEVICE_VISIBLE|HOST_COHERENT, inside a process that already has the device, queues and ~41
threads live, and then whatever the scheduler does before it reports progress. So the hang is in the layer between
"pin the mapping" and "the load proceeds", not in the pin.
One sub-question my own harness lost: the control arm with the fleet's no-SVM trio (HSA_USE_SVM=0 …) failed, as in my
first attempt (0x1000 HSA_STATUS_ERROR), but this run's detail line was swallowed by my own grep filter, so the trio's
role is consistent with attempt 1 rather than freshly measured. It matters only if someone runs the import with the trio
set — the engine's default does not.

The next test, if the owning lane wants it

Same probe, two changes: pin with hsa_amd_memory_lock_to_pool against the pool/flags this path uses, and run it inside
a process that has already opened the device (one queue created, a few threads alive). If that still returns in seconds, the
stall is in the runtime's own import bookkeeping or in the caller's wait, and the next place to look is
access_policy.c/allocator.c's import state machine and the buffer_submit_and_wait path — not the driver.

Nothing here changes the issue's status or its mitigation; #1343 keeps the import opt-in either way. Numbers and the probe
source stay in ~/amd-oss/hrx-study/evidence/fabric/lockprobe-20261107T1928b/ on strixhalo.

release-check: measure the HRX weight placement (engine #1339), not t…

docs: stop citing the deleted import knob as the Zyphra page's eviden…

Four arms on the isolated call: a 16.4 GiB pin finishes in 0.13 s / 0.51 s, so the stall is above ROCr — plus two of my own arms were wrong, and one axis the source settles without measuring

Consolidating my standalone-probe work here (previous comments: #1,
#2). This is one lane's independent control, not a claim
on the issue; I ran it because the userspace trace cannot separate "one long pin" from "a lock that never converges", and the answer
turned out to be neither.

Probe: ~/wt/fabric/lockprobe.c (dlopens the shipped libhsa-runtime64.so.1, no ROCm headers), mmap the GGUF
MAP_SHARED|PROT_READ, pre-fault single-threaded, then time one lock. Module F87D0BDF248067862F83BA3, kernel
7.4.0-rc1-1bit, one gpu-hold ticket per arm.

arm	call	range	result
1	`hsa_amd_memory_lock`, with the fleet's no-SVM trio	16.4 GiB	pre-fault 34.6 s (484 MiB/s, cold); lock refused in 0.027 s, `0x1000 = HSA_STATUS_ERROR`
2	`hsa_amd_memory_lock`, no trio	16.4 GiB	lock 0.128 s, `rc=0`
2b	same, anonymous mapping	16.4 GiB	lock 4.206 s, `rc=0`
2c	same, +2 threads re-faulting pages throughout	16.4 GiB	lock 2.058 s, `rc=0`
3	`hsa_amd_memory_lock_to_pool` on the GPU agent's pools	16.4 GiB	all three pools: `0x28` in 0.000 s
3b	`hsa_amd_memory_lock`, second ticket, colder cache	16.4 GiB	pre-fault 17.3 s (970 MiB/s); lock 0.512 s, `rc=0`
4	`hsa_amd_memory_lock_to_pool` on the CPU agent's pools	1 GiB	never ran — my probe looked for `devtype == 2`, which is `DSP`; `hsa.h` has `HSA_DEVICE_TYPE_CPU = 0`. Fixed, arm re-queued

What this clears

"One 16 GiB pin is inherently slow or never converges" is false on this box: 0.128 s and 0.512 s on two independent
tickets, and the anonymous mapping is slower (4.2 s) than the file-backed one — the opposite of a file-pages problem.
"Notifier invalidations restart the range and it never converges" is false at this layer: arm 2c keeps two threads
faulting pages for the whole call and it still returns in 2.1 s.
0x28 has a name: HSA_STATUS_ERROR_INVALID_MEMORY_POOL = 40 (hsa_ext_amd.h:626), returned instantly for every
GPU-agent pool — the same header documents the parameter as "Global memory pool owned by a CPU agent".

What the source settles without a measurement

The import never asks for a device pool, in three hops:
allocator.c:1826-1828 forces type |= HOST_LOCAL | HOST_COHERENT | DEVICE_VISIBLE; :741-752 sees
requires_host_local and not requires_device_local and therefore picks memory_pools.host_fine[]; :967-973 builds
host_fine[i] from system->host_memory_pools[host_ordinal].fine_pool — a CPU agent's FINE_GRAINED pool. So the pool swept
in arm 3 was the wrong agent by construction, and arm 4 (CPU fine pool) is the shape the runtime really uses.

Where that leaves the hang

Every layer below the runtime's own import bookkeeping now has a fast, repeatable control: pinning is milliseconds, concurrent
faults do not break it, file vs anonymous does not matter, and the pool question is a CPU-agent pool. The remaining candidates are
inside hrx_allocator_import_buffer's state machine (or in whoever waits on it) — not in ROCr, KFD or the driver. The next probe
worth running is that function itself on a 16.4 GiB host pointer inside a process with the device already open; I have queued a
variant of it, and if it shows the same fast path then the search should move to the caller's wait, not to the pin.

Two of my six arms measured my own mistakes (HSA_USE_SVM=0 refuses an SVM-class API; wrong agent's pools), which is why I am
reporting the table rather than a conclusion: the useful output here is which layers are cleared and by what number.
Nothing in this comment changes the issue's status or its mitigation — GGML_HRX_WEIGHT_HOST_PLACEMENT staying opt-in is still
the right default until someone finds where the 11 minutes go.

Closing my arc here: the pool call is validated, and the one combination left unmeasured is named

Last arm for me on this issue (the previous comment has the full table and the source proof). It ran the small branch on purpose —
the box has been at 5-7 GiB free for hours, and the adaptive ticket was written to take the 1 GiB validation rather than thrash a
16.4 GiB pre-fault inside someone's GPU turn. inside-hold MemAvailable: 5 GiB, token held 13 s, die 62 °C.

arm	call	result
control	`hsa_amd_memory_lock`, 1 GiB file mapping	pre-fault 12.2 s (84 MiB/s, cold), lock 0.109 s, `rc=0`
GPU agent's pools (control for the error)	`lock_to_pool`, 1 GiB	3 pools, all `0x28 INVALID_MEMORY_POOL` in 0.000 s
CPU agent's pools	`lock_to_pool`, 1 GiB	4 pools, all `rc=0` in 0.000 s, including the `flags=0x2` FINE_GRAINED one

That closes the instrument question: lock_to_pool against the CPU agent's fine pool — the pool allocator.c:967-973 builds
host_fine[] from, and the one the import actually passes at :741-752 — succeeds and is instant. The two earlier failures were
mine (no-SVM trio refusing an SVM-class API; GPU-agent pools), not properties of the driver.

What remains unmeasured, stated so nobody reads this as more than it is: the combination 16.4 GiB × CPU-fine pool. Its two
factors are both measured and both fast — a 16.4 GiB plain lock takes 0.128 s / 0.512 s on two independent tickets, and the CPU
fine pool accepts instantly — but the product is not, because it needs ~16 GiB resident and this box has not had 40 GiB free for
hours. If anyone with headroom wants it, ~/wt/fabric/runLockprobe7.sh on strixhalo takes the full branch automatically when
MemAvailable >= 40 GiB and is otherwise a no-op.

Net for this issue, from my side: four layers have a repeatable fast control (pin, file-vs-anon, concurrent faults,
pool), and the pool involved is a host pool by construction. So the 11 missing minutes are not in pinning, not in the
notifier, not in the pool choice, and not in hsaKmt/KFD — they are in hrx_allocator_import_buffer's own state machine or in
whoever waits on it. That is where I would look next, and it is cheaper to look there than to keep measuring below it.

Navigation Menu

Repository navigation

hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339

hrx: the weight-mapping import (fork #226) stalls model load above a size threshold on the shipped pin `a3e8d42c9`

Evidence

Workaround

Pending

The 27B import arm hangs, it is not just slow (1 Hz trace)

Proof at the pin `092eb9dbc` (fork #231 as merged, tip of `1bit/hrx-vulkan-patched`)

Root cause: the import write-faults a shared file mapping, and page-cache writeback keeps restarting it

What would fix it

A code-level lead from another lane, and one inference in the trace comment worth re-checking

Where the import actually goes (userspace, all in trees I have checked out)

The inference I would re-check: flat `drm-total-gtt` is not obviously "nothing is happening"

Cross-reference

The pin is not the hang: one `hsa_amd_memory_lock` over the same 16.4 GiB mapping finishes in 0.13 s

What was run

What that rules out, and what it does not

The next test, if the owning lane wants it

Four arms on the isolated call: a 16.4 GiB pin finishes in 0.13 s / 0.51 s, so the stall is above ROCr — plus two of my own arms were wrong, and one axis the source settles without measuring

What this clears

What the source settles without a measurement

Where that leaves the hang

Closing my arc here: the pool call is validated, and the one combination left unmeasured is named

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: with GGML_HRX_WEIGHT_HOST_PLACEMENT=1 the weight-mapping import stalls the load of large files (16.4 GB dense, 76 GB): no error, no progress #1339

Description

hrx: the weight-mapping import (fork #226) stalls model load above a size threshold on the shipped pin a3e8d42c9

Evidence

Workaround

Pending

Activity

bong-water-water-bong commented on Nov 7, 2026

The 27B import arm hangs, it is not just slow (1 Hz trace)

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

Proof at the pin 092eb9dbc (fork #231 as merged, tip of 1bit/hrx-vulkan-patched)

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

Root cause: the import write-faults a shared file mapping, and page-cache writeback keeps restarting it

What would fix it

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

A code-level lead from another lane, and one inference in the trace comment worth re-checking

Where the import actually goes (userspace, all in trees I have checked out)

The inference I would re-check: flat drm-total-gtt is not obviously "nothing is happening"

Cross-reference

bong-water-water-bong commented on Nov 7, 2026

The pin is not the hang: one hsa_amd_memory_lock over the same 16.4 GiB mapping finishes in 0.13 s

What was run

What that rules out, and what it does not

The next test, if the owning lane wants it

bong-water-water-bong commented on Nov 8, 2026

Four arms on the isolated call: a 16.4 GiB pin finishes in 0.13 s / 0.51 s, so the stall is above ROCr — plus two of my own arms were wrong, and one axis the source settles without measuring

What this clears

What the source settles without a measurement

Where that leaves the hang

bong-water-water-bong commented on Nov 8, 2026

Closing my arc here: the pool call is validated, and the one combination left unmeasured is named

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

hrx: the weight-mapping import (fork #226) stalls model load above a size threshold on the shipped pin `a3e8d42c9`

Proof at the pin `092eb9dbc` (fork #231 as merged, tip of `1bit/hrx-vulkan-patched`)

The inference I would re-check: flat `drm-total-gtt` is not obviously "nothing is happening"

The pin is not the hang: one `hsa_amd_memory_lock` over the same 16.4 GiB mapping finishes in 0.13 s