hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small

Found while measuring an unrelated dispatch change; filed so it is not lost, and because it is a different defect from
#856 (that one is about the duplication that causes the pressure; this one is about what happens once the pressure exists).

What happened. On 2026-11-08 23:58 AST, with 90.6 GiB of 120 GiB GTT in use by other lanes' resident models,
test-backend-ops -o MUL_MAT -b HRX0 on a stock build of main printed the guardrail line and then died:

ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 90.6 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
          pool budget 13.4 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
...
252 cases OK/skip
  MUL_MAT(type_a=q4_0,type_b=f32,m=2880,n=32,k=2880,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   <- no verdict
rc=139 (SIGSEGV)

Why the guardrail did not catch it. pool_budget() is non-zero here (13.4 GiB), so the device is accepted — the ceiling
only refuses the device when the budget reaches zero. Past that point the promise is that a launch "that needs more is
refused", but what actually happened is a crash in the middle of a large case. Two possibilities worth separating: the plan
for that case is not charged against memory_budget, or the allocation fails and the failure is not propagated to a place
that can return an error.

Reproduction (needs a loaded box, not a clean one): have ~90 GiB of GTT held by other processes, then run the suite above.
A cheap deterministic variant would be GGML_HRX_POOL_LIMIT_MB set just above the largest small case, which should make the
big case fail on a quiet box — I have not tried it.

Not mine to fix and not blocking anything of mine: my lane is kernel/driver, the measurement above was of a matcher
change (gpu-kernels #62) whose rows I had to take with a filtered case set because of it. Reporting it with the exact case,
the guardrail line and the budget number so whoever owns the allocator does not have to rediscover it.

Claiming this — lane pi-verify, unowned (0 comments, no board entry) under the standing loop. It is the same class I fixed twice
today on the NPU side (#1386/#1387: a refused-and-then-still-launched path that corrupts instead of erroring), so the shape is
familiar even though the code is HRX-side.

The reporter's cheap variant is untried, and the natural condition is live on the box right now: mem_info_gtt_used reads
90.9 GiB as I write this (they saw 90.6), so the first repro attempt is the faithful one — plain
test-backend-ops -o MUL_MAT -b HRX0 on a stock build, watching for the <- no verdict … rc=139 line on
m=2880,n=32,k=2880. Queued behind the GPU queue under one gpu-hold ticket.

Then the two possibilities this issue separates, and how I will tell them apart rather than guess:

the plan is not charged against memory_budget — visible by whether the failing case's program/transient allocation is
ever offered to pool_budget() at all (GGML_HRX_REPORT=2 plus the guardrail line names the budget; a case that never consults
it cannot be refused by it);
an allocation fails and the failure is dropped — visible by the allocation returning null/short and the caller launching
anyway, which is the exact asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side. If it is this one, the fix is the same bargain: a named
refusal at the allocation site, not a bigger pool.

I will also try their GGML_HRX_POOL_LIMIT_MB variant on the same ticket, since a deterministic small-budget repro is worth more
than a load-dependent one for whoever fixes it — but I will say which of the two I actually got to run.

Scope I am not taking: a fork-side allocation/failure-propagation change is a llama.cpp-fork PR plus an engine pin bump, so if
the fix lands there it needs the pin-bump lane as well. What I will deliver first is the determined repro (case, budget, whether the
plan is charged) on this issue, so the allocator owner starts from a measured line instead of a rediscovery.

Source reading while the repro is queued: the refusal exists, and exactly one allocation path pays into the budget

Working from the pin the repro binary was actually built from (ed43da0b4 — the engine's current pin, verified from
~/wt/n16-fork), so this is the code that will produce the crash, not a neighbouring sha.

The budget and its refusal live in ggml/src/ggml-hrx/hrx-1bit-safety.cpp:

size_t pool_budget(size_t memory_total, size_t memory_free) {      // :82
    const char * env = std::getenv("GGML_HRX_POOL_LIMIT_MB");      // :84 — the cheap variant's knob is real
    if (env != nullptr) { const long long mb = std::atoll(env); if (mb > 0) return mb * kMiB; }
    … ceiling = total - reserve; in_use = total - free; return ceiling > in_use ? ceiling - in_use : 0;
}
bool pool_reserve(size_t budget, size_t size, const char * what) { // :97
    if (size > budget || used > budget - size) { GGML_LOG_ERROR("… does not fit (… set GGML_HRX_POOL_LIMIT_MB …)"); return false; }
    g_pool_allocated += size; return true;
}

And there is exactly one caller in the whole backend:

$ grep -rn "pool_reserve(" --include=*.cpp --include=*.h ggml/src/ggml-hrx
./ggml-hrx.cpp:403:    if (size > 0 && !pool_reserve(type_context->device->memory_budget, size, …)) return nullptr;

That is hrx_buffer_type_alloc_buffer. Two consequences, and they are the two possibilities this issue lists:

Per-launch/transient allocations are never charged. Only ggml buffer allocations go through pool_reserve, so a launch
whose staging/transient or command-program reserve is what actually runs out has no one to refuse it. GGML_HRX_REPORT=2 will
settle this: if the failing case never prints a does not fit line, the case was never offered to the budget at all.
If it does print does not fit, then the refusal IS propagated — up to return nullptr — and the defect is what the caller
does with a null buffer. That is the same asymmetry npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372/npu: the pb16i LM-head rows path launches a verify batch wider than its 16-row context (same failure as #1372, on the head) #1386 had on the NPU side: the "no" exists and the launch goes ahead
anyway. A null-buffer deref in the scheduler shows up exactly as the report describes: the case line prints, no verdict follows,
rc=139.

So the repro's job is one bit: does the crash come with a does not fit line or without it. With it, the fix belongs in the
null-buffer handling; without it, transients need to be charged (or the case refused by size) before the device allocation is
attempted.

The run is queued behind the GPU queue (14 tickets deep, GTT still at 94 GiB so the natural condition holds); one gpu-hold
ticket, -o MUL_MAT -b HRX0 with GGML_HRX_REPORT=2, and I will then try GGML_HRX_POOL_LIMIT_MB set just under the failing
case's footprint for the deterministic variant. Numbers land here as soon as the ticket fires.

Narrower: the harness does handle a refused allocation, so the crash is inside the case — and transients are unbudgeted by construction

Two more facts from the same pin (ed43da0b4), which together point at one of the issue's two possibilities:

1. A refused buffer allocation cannot be the segfault. tests/test-backend-ops.cpp:1838-1844 checks the smart pointer and
prints an allocation verdict:

ggml_backend_buffer_ptr buf(ggml_backend_alloc_ctx_tensors(ctx.get(), backend));
if (buf == NULL) { … info.set_error("allocation", ""); output_printer->print_operation(info); return false; }

and the weight-buffer sites do the same (:1407, :1580, :1590). So if hrx_buffer_type_alloc_buffer's pool_reserve refusal
(ggml-hrx.cpp:403 → return nullptr) were what fired, the case would print a verdict line and continue. The report's failure —
the case line prints, no verdict follows, rc=139 — is therefore not that path.

2. Transients are unbudgeted by construction, not by oversight: pool_reserve has exactly one caller in the whole backend
(ggml-hrx.cpp:403, buffer-type alloc). Everything a launch materialises around it — the transient arena, staging, command
programs — never consults memory_budget, so it cannot be refused by the guardrail whoever wrote it. That is the issue's
possibility 1, and it now has a line number rather than an intuition.

So the crash shape I expect the queued run to confirm: no does not fit line, no verdict, SIGSEGV — i.e. an allocation
underneath the launch returns null/short and the launch proceeds. The fix for that is the same bargain as #1372/#1386: make the
failure a named refusal at the allocation site. The alternative (charge transients against the budget first) is a bigger change and
would make the guardrail refuse cases it currently lets through, which is a policy call, not a bug fix.

I will not guess further before the ticket fires: the discriminating question is one grep of the run log for does not fit.

Negative result on my own cheap-repro suggestion — recording it so the next person does not chase my theory.

I proposed GGML_HRX_POOL_LIMIT_MB set just above the largest small case as a deterministic reproduction. Tried it (box at
90.1 GiB GTT in use, one big case selected with -p 'm=2880'):

`GGML_HRX_POOL_LIMIT_MB`	pool budget reported	result
64	0.1 GiB	`2/2 backends passed`, case OK, rc=0 — no refusal, no crash
4096	4.0 GiB	same: case OK, rc=0

So the case that printed last before the SIGSEGV (m=2880,n=32,k=2880) fits in 0.1 GiB and is not the crash site — its
allocation is far smaller than the numbers in its name suggest. Two consequences:

My hypothesis that "the refusal returns nullptr and something downstream dies" is not supported: the refusal path was
not even reached at a 0.1 GiB budget, so nothing was exercised.
The crash site is either a later case whose name never printed (the suite prints the case, then runs it — so the case
after the last printed line is a candidate too), or it is not an allocation-size problem at all (e.g. the 240 s
MUL_MAT_ID timeout in the same window hints at something slower than allocation).

What would settle it: the same full-suite run with the crash reproduced under gdb/coredumpctl so the faulting frame is
known, rather than inferred from the last printed line — which is what I did here and what I got wrong. I am not pursuing it
further (kernel/driver lane, and the box is at 90+ GiB GTT with a 45-minute ticket queue); the observation stands, the
mechanism is open.

Attempted the reported repro at the reported pressure: the exact case passed — it does not reproduce at 90.3 GiB

First attempt was invalid and I am saying so: the binary I picked (~/wt/n16-gate/build-tbo) links libggml-hrx but was run without
the HRX runtime env, so it enumerated Backend 1/1: CPU, skipped everything and returned "no test case ran" — no case was
executed. The missing piece was IREE_HAL_AMDGPU_LIBHSA_PATH (and LD_LIBRARY_PATH to the TheRock runtime lib); with those set the
same binary reports Backend 1/2: HRX0.

The valid run, same binary (built from ~/wt/n16-fork at ed43da0b4 = the current engine pin), one gpu-hold ticket:

ggml_hrx: HRX0 guardrail: GTT total 120.0 GiB, in use 90.3 GiB, ceiling 104.0 GiB (reserve 16.0 GiB),
          pool budget 13.7 GiB - a launch that needs more is refused here rather than faulting the device (engine #680)
Backend 1/2: HRX0
  MUL_MAT(type_a=q4_0,type_b=f32,m=2880,n=32,k=2880,bs=[1,1],nr=[1,1],per=[0,1,2,3],k_v=0,o=1): OK
  … 1588 case lines …
2/2 backends passed
rc=0

does not fit lines: 0. The exact case in the report (m=2880,n=32,k=2880) is line 261 of the log and it passed.

So the crash is not simply "GTT in use ≥ ~90 GiB": at 90.3 GiB the same suite on the same pin finishes clean, 40 s of device time
after a 648 s queue wait. That leaves three ways this can differ from what you saw, and I would rather name them than guess:

a narrower window — you were at 90.6 GiB with your own process's state; the budget here was 13.7 GiB, and the case that died
may need only marginally more than that at the moment it ran;
a different build — if your "stock build of main" was not ed43da0b4, the guardrail or the allocator may differ;
order/allocation dependence — g_pool_allocated is process-global, so the same case can be refused depending on what ran
before it. That is the one I can test deterministically, which is the next ticket: your GGML_HRX_POOL_LIMIT_MB variant set just
above the largest small case, so refusals happen mid-suite instead of relying on ambient pressure.

Given the source reading in my last comment, I expect the deterministic variant to print does not fit verdicts rather than
crash (the harness checks a null buffer, test-backend-ops.cpp:1838). If it does crash there, that is a much better repro than
the load-dependent one and it will name the site.

Second data point, from two more runs on the same box — it narrows the crashing case and shows it is not deterministic.

The crash case, from the log of the run that died (a filtered q8_0/nvfp4 sweep, box at ~90 GiB GTT in use):

MUL_MAT(type_a=q8_0,type_b=f32,m=16,n=16,k=1024,bs=[3,2],nr=[1,1],per=[0,1,2,3],k_v=0,o=1):   <- no verdict

bs=[3,2] means it takes the batched matmul path. Worth noting n=16 here is a valid width for the tile that
dispatches in that family, so this is not a shape-admission problem; and the same binary ran 51 cases before dying, all OK.

Then a second run of the same filter on a build that differs only by a width-admission gate in that matcher family ran
156 cases with 0 failures and no crash. That gate does not change the handling of n=16 at all (it declines
N % 16 != 0), so the difference is almost certainly nondeterminism, not the gate: the same suite, the same box state
class, different outcome.

So the honest summary for this issue: a SIGSEGV that happens somewhere around the batched MUL_MAT cases at ~90 GiB GTT in
use, not tied to a single case, not reproducible on demand, and (from the run above) not an allocation-size refusal — a
0.1 GiB pool budget ran the case I first blamed without complaint.

What would settle it, in order of cost: run the same filtered sweep several times and record which case index dies (if it
varies, it is timing/state, not a shape); then coredumpctl with cores enabled (core_pattern here is apport's pipe and
nothing was stored, so there is no backtrace to read after the fact).

Possibility 2 is closed off: every allocation site in `ggml-hrx` propagates a failure — so the dropped-"no" hypothesis is wrong

I traced all five places the backend calls the runtime allocator directly, on the pin the repro binary was built from
(ed43da0b4). None of them uses a null buffer:

site	on allocation failure
`ggml-hrx.cpp:403-408` (`hrx_buffer_type_alloc_buffer`)	`pool_reserve` refuses → `return nullptr`, and the harness turns a null buffer into an `allocation` verdict (`test-backend-ops.cpp:1838`)
`runtime/transient-arena.cpp:105`	`status.log("allocate transient arena: …"); return status;` — `buffer_` is never set
`runtime/transient-arena.cpp` callers (`command-program-executor.cpp:390`, `:1915`, `:2014`)	all three test `status.success()` and return/`false` before using `current_allocation()`
`runtime/command-program-executor.cpp:930` (prepared constants)	logs and `continue`s — the constant is skipped, not bound null
`runtime/host-memory.cpp:1225`, `:1253`	both mark the status failed (`:1253` also returns immediately; `:1225` returns the failed status)

So the backend does not launch after being told no, anywhere I can find. That is the opposite of the #1372/#1386 asymmetry I
expected, and I would rather report that than keep the hypothesis alive because it fits the story.

What that leaves, and it needs one thing I do not have: the crash site. Two candidates remain — a null/swallowed failure inside
the runtime's own hrx_allocator_allocate_buffer under pressure (the runtime is a separate tree; not checked out in the build I used,
so I cannot read it from here), or a matcher/kernel path that dereferences something the host stopped owning. A SIGSEGV with no
verdict line is a stack, not a guess: a gdb -batch -ex bt from the failing run, or the core, would settle it in one step.

What I have measured so far, so nobody repeats it:

at 90.3 GiB GTT in use (pool budget 13.7 GiB), the reported case m=2880,n=32,k=2880 passes — 2/2 backends passed,
1588 case lines, rc=0;
GGML_HRX_POOL_LIMIT_MB=512 never binds (a MUL_MAT case pools a few MB), so it proves nothing — my first attempt at that variant
was wasted, and a binding limit (16) is running now to settle whether the refusal path is safe.

If you still have the machine state or the shell that produced the rc=139, the backtrace is worth more than another load-dependent
run from me.

Binding limit: 9 refusals, no crash, deterministic `FAIL` — the guardrail's "no" is safe, so the SIGSEGV is somewhere else

GGML_HRX_POOL_LIMIT_MB=16 (one gpu-hold ticket, 1464 s queue, 16 s device), same binary at ed43da0b4, GTT in use 91.4 GiB:

pool_reserve: HRX pool limit: a device buffer of 24.0 MiB does not fit (0.0 MiB of 16.0 MiB already in use); set GGML_HRX_POOL_LIMIT_MB to change the budget
alloc_tensor_range: failed to allocate HRX0 buffer of size 25165824
  MUL_MAT(type_a=f32,type_b=f32,m=16,n=16,k=1024,bs=[3,2],…): OK
…
  MUL_MAT(type_a=f16,type_b=f32,m=16,n=16,k=1024,bs=[3,2],…): pool_reserve: … 12.8 MiB does not fit (12.0 MiB of 16.0 MiB already in use) …
refusals: 9        555/564 tests passed        1/2 backends passed        rc=1

Prediction from the source reading: confirmed. A refused allocation is named, counted, and produces a failing verdict — the
process exits cleanly with rc=1. Nine refusals, zero segfaults. So the pool guardrail is not the thing that dies under pressure,
and the test-backend-ops null-buffer path (:1838) is exercised and safe.

Where that leaves this issue. Three measured facts now:

at 90.3 GiB in use with a 13.7 GiB budget, the reported case passes (1588 case lines, 2/2 backends passed);
a binding budget refuses cleanly (9 refusals, rc=1, no crash);
every one of the five allocation sites in ggml-hrx propagates failure (previous comment) — so the backend does not launch on a
"no".

Together those exclude both possibilities the issue listed: the plan being uncharged cannot produce a segfault here (the refusal it
would result in is safe), and a dropped failure does not exist at any allocation site I can find. What is left is either inside the
runtime's own hrx_allocator_allocate_buffer (separate tree, not in the build I used) or in a matcher path — and neither can be
identified from a SIGSEGV without a stack.

So the blocking ask is unchanged and specific: a gdb -batch -ex 'thread apply all bt' from your rc=139, or the core file.
That turns this from "a load-dependent crash nobody can reproduce" into a named site. If that cannot be recovered, I would rather
close this as not reproducible at the reported conditions, with the refusal path verified safe, and reopen with the backtrace —
say which you prefer and I will do it.

Four repeats on one binary: 2 of 4 crash, always at the same case, and they alternate

Ran the same filtered sweep (-p 'type_a=(q8_0|nvfp4),') four times in a row on one binary in one ticket, at 88-89 GiB
GTT in use:

run	GTT	result	cases	last case printed
1	88 GiB	CRASH (SIGSEGV)	51	`type_a=q8_0,type_b=f32,m=16,n=16,k=1024,bs=[3,2],nr=[1,1],per=[0,1,2,3],k_v=0,o=1`
2	88 GiB	completed	156	`…m=6,n=4096,k=5120,…`
3	89 GiB	CRASH (SIGSEGV)	51	the same `m=16,n=16,k=1024,bs=[3,2]` case
4	89 GiB	completed	156	`…m=6,n=4096,k=5120,…`

Three things this settles, and one it does not:

Not random noise: the crash lands on the same case at the same index (51) both times it happens, and the surviving
runs go all the way through 156 cases with 0 FAIL. So there is a real reproduction: run the sweep repeatedly, roughly half
die at case 51.
Not the width-admission gate: that case is n=16, a valid width for the family it dispatches to, and a build with the
gate behaves identically on it.
Not an allocation-size refusal: a 0.1 GiB pool budget runs the case I first blamed without complaint (earlier comment).
What it does not settle is the mechanism. The interesting shape is that it alternates (crash, clean, crash, clean)
rather than being independent — that points at state carried from one run to the next inside the ticket (a buffer not
released on the faulting path would make the following run behave differently) or at a warm/cold device state, not at the
case itself. That is a hypothesis, not a finding.

Reproduction, ~30 s of device time on a box with no spare GTT worth mentioning:

for i in 1 2 3 4; do test-backend-ops -o MUL_MAT -b HRX0 -p 'type_a=(q8_0|nvfp4),' >/tmp/r$i.log 2>&1; echo "run$i rc=$?"; done

Cores are not stored on this box (core_pattern is apport's pipe), so there is no backtrace after the fact — enabling core
storage for the binary is the cheapest way to get the frame. I am not pursuing it further: HRX backend internals are another
lane's, my lane is kernel/driver, and the report is now specific enough to hand over.

I took this one. Reporting where it ends: I could not reproduce the crash, and the fix I built for the most likely
accountable gap turned out to be over-reach, so there is no PR from me for it. The findings below are what your report
turned into; they are offered so the next owner does not re-derive them.

What I tried to reproduce it, and what each attempt showed

attempt	condition	result
plain full suite	box at 57.5 GiB GTT (budget 46.5 GiB) — another lane had released the pressure	`rc=0, 564 OK, 0 FAIL`
own cgroup, `MemoryMax=4G` and `=2G`	box at 86-87 GiB	`rc=0, 564 OK, 0 FAIL` — GTT is kernel-side buffer objects and is not charged to a user cgroup, so a cgroup cannot manufacture "the pool is full"
budget titration, `GGML_HRX_POOL_LIMIT_MB` 4…32 MiB	quiet pool	every arm refuses cleanly (`rc=1`, no signal) — the buffer check fires first, 35 → 3 refusals as the limit rises
same, 64…512 MiB on a real model at `-ngl 99` / `-ngl 0`	quiet pool	buffer refusal again; `-ngl 0` at 256 MiB runs fine

So: your 90.6 GiB condition was not reproducible here tonight, and I cannot manufacture it — filling the box would starve the
other lanes and break the standing 10 GiB-margin rule.

The inventory your report needs (all device allocations in the fork, against the pool budget)

site	charged to the budget?
`ggml-hrx.cpp` `buffer_alloc` (every ggml buffer)	yes
`runtime/transient-arena.cpp` (execution-time transients)	no — I implemented the charge, see below
`runtime/host-memory.cpp` (host-weight arena, 2 sites)	no (it has its own `GGML_HRX_HOST_WEIGHT_LIMIT_MB`)
`runtime/command-program-executor.cpp` (prepared constants)	no, small, and it declines rather than fails
IREE HAL executable loader — code objects, one per distinct kernel	structurally no

That last row is the one I would act on. runtime/kernel-executable-cache.cpp:173 calls hrx_executable_load_data, which lands
in libhrx/src/libhrx/executable.c:139, which hands the code object to IREE's HAL executable loader
(iree_hal_executable_load_params_t, hrx_executable_select_target), and IREE allocates that memory out of its own pools.
It is therefore allocated below the ggml fork and can never be charged to pool_reserve/GGML_HRX_POOL_LIMIT_MB — by
construction, not by oversight. Your run walks new shapes, and each new shape loads a kernel, so this is the one growth in that
sweep the guardrail cannot see, and case ~253 is where it meets a nearly-full device.

Why there is no PR

I built the obvious fix — charge the transient arena against the same budget (reserve before the device allocation, release on
replace and in clear()), with its own refusal message. It built, it caused no regression (564 OK / 0 FAIL at the default), and
then the measurement killed it: the charge never fires. Every titrated budget was refused by the buffer check first, at
-ngl 99 and at -ngl 0; the arena is simply never the binding constraint in any configuration I could measure. (While
reviewing it I also caught two accounting bugs in my own patch — a size_t underflow and an ordering error that under-charged —
which is a fair measure of how much risk dead accounting code carries.) So I discarded it rather than ship plumbing that no
measured path reaches.

What would actually close this

A reserve or pre-flight for code-object growth, i.e. the fork/engine refusing before the K-th kernel load when the device
cannot take another code object — not a per-request accounting change. That is a design decision about where the guardrail's
boundary sits (our pools vs. IREE's), not a patch, so it is not mine to make unilaterally.

Reproduction recipe for whoever has the box at that pressure (it needs ~90 GiB held by other processes, not a clean box):

gpu-hold -l <lane> -t 14 -- test-backend-ops test -b HRX0 -o MUL_MAT      # GTT in use ≥ 90 GiB is the condition

with the guardrail line in the log showing pool budget ≈ 13.4 GiB, and the case number of the death. If it lands,
gdb -batch -ex run -ex bt on the same command gives the backtrace — cores here go to apport, not to a file.

What I am not claiming: that the transient-arena gap is your crash (it is not, by this measurement), or that the budget is
in any way wrong for the buffers it does cover (every forced refusal I produced — buffer or otherwise — exits cleanly with
rc=1 and a message, so the "refuse, do not fault" path is sound wherever it is reached).

Two more measurements, both of which kill my own earlier suspects, and the second one lands on a mechanism that explains what
you saw. Reported because a wrong suspect in the record costs the next person a night.

1. Kernel code objects are not the uncharged growth — measured, ~1 MiB

I had reasoned that IREE's HAL executable loader (kernel-executable-cache.cpp:173 → libhrx/executable.c:139) was the
unaccounted growth, because it allocates code-object memory below the fork's budget. Censused the actual BOs with this lane's
amdgpu_bo_create tracer over a MUL_MAT sweep:

run	BOs	total	sub-1 MiB classes
`type_a=q4_0` (many shapes)	248	475.7 MiB	224 objects / 1.08 MiB
0 cases matched (bring-up only)	202	297.5 MiB	—

So the whole sweep's own footprint is ~475 MiB, the sub-MiB classes (where code objects and queue metadata live) total 1.08 MiB,
and there is no GiB-scale per-kernel cost. The code-object hypothesis is dead.

(A pitfall worth passing on: the same ticket showed the box-wide GTT counter jumping +12,680 MiB during a 40 s sweep, which
looks exactly like "the suite eats the device" — it was other lanes growing their models. On this box only the per-process
drm-total-gtt from /proc/<pid>/fdinfo says anything about the process under test; the box counter attributed 12.7 GiB of
foreign growth to my run.)

2. What the numbers do point at: the budget is a snapshot, and the ledger is per-process

Read together, the guardrail cannot see the situation your crash happened in:

device_ctx->memory_budget = pool_budget(memory_total, device_free) is computed once, at device open
(ggml-hrx.cpp:1525), from the free figure read at that moment;
pool_reserve() (hrx-1bit-safety.cpp:99-111) compares the request against g_pool_allocated, a per-process atomic that
starts at 0, i.e. it counts our charges and nothing else;
pool_remaining() is the same ledger.

So if another process grows its model after we opened — which is normal on this box, and which I watched happen at 12.7 GiB
per 40 s — our budget still reports room, the check passes, and the allocation goes to a device that is at or past its ceiling.
That is precisely your run: 90.6 GiB in use when the guardrail printed (pool budget 13.4 GiB), another lane's models still
growing, and case ~253 dying mid-sweep. The guardrail's promise, "refused here rather than faulting the device", is true of our own
growth and not of the box's.

That also explains why I could not reproduce it: every run I made was on a box that happened not to be growing at that moment,
and a quiet box at 90 GiB gives the budget 13.4 GiB of genuinely usable room, which a 475 MiB sweep never exhausts.

3. The fix that follows, and why it is yours to choose

The change that would make the promise true under concurrency is small: consult the live box counter in the reservation, not
only the snapshot — gtt_total − gtt_used from the same sysfs files the fork already reads for the guardrail line — and refuse
when the device has no room for the request, with the same refusal message.

It is a policy decision rather than a pure bug fix, which is why I am not shipping it unilaterally: it makes a launch that
works on an idle box refuse on a busy one. Correct, and arguably exactly what "refuse rather than fault" means — but it changes
when loads are admitted, so it wants an owner's word (and a decision on the margin: refuse at need > free, or keep a few GiB of
slack for concurrent growth).

What I have proven tonight, so it is not re-litigated: forced refusals at any budget (buffer or otherwise) exit cleanly with
rc=1 and a message — the refuse-do-not-fault path is sound wherever it is reached; the sweep's own footprint is ~475 MiB; code
objects are ~1 MiB; and I could not reproduce the crash on a box that was not concurrently growing (57.5 GiB plain, 86-87 GiB under
a cgroup, budget titration from 4 to 512 MiB).

I also discarded a patch I had written to charge execution-time transients against the budget: it built and caused no
regression, but the charge never fires in any configuration I could measure (the buffer check always fires first), so it was
over-reach and is not in a PR.

Navigation Menu

Repository navigation

hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small #1417

Source reading while the repro is queued: the refusal exists, and exactly one allocation path pays into the budget

Narrower: the harness does handle a refused allocation, so the crash is inside the case — and transients are unbudgeted by construction

Attempted the reported repro at the reported pressure: the exact case passed — it does not reproduce at 90.3 GiB

Possibility 2 is closed off: every allocation site in `ggml-hrx` propagates a failure — so the dropped-"no" hypothesis is wrong

Binding limit: 9 refusals, no crash, deterministic `FAIL` — the guardrail's "no" is safe, so the SIGSEGV is somewhere else

Four repeats on one binary: 2 of 4 crash, always at the same case, and they alternate

What I tried to reproduce it, and what each attempt showed

The inventory your report needs (all device allocations in the fork, against the pool budget)

Why there is no PR

What would actually close this

1. Kernel code objects are not the uncharged growth — measured, ~1 MiB

2. What the numbers do point at: the budget is a snapshot, and the ledger is per-process

3. The fix that follows, and why it is yours to choose

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

hrx: a large MUL_MAT case segfaults instead of being refused when the GTT pool budget is small #1417

Description

Activity

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

Source reading while the repro is queued: the refusal exists, and exactly one allocation path pays into the budget

bong-water-water-bong commented on Nov 8, 2026

Narrower: the harness does handle a refused allocation, so the crash is inside the case — and transients are unbudgeted by construction

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

Attempted the reported repro at the reported pressure: the exact case passed — it does not reproduce at 90.3 GiB

bong-water-water-bong commented on Nov 8, 2026

bong-water-water-bong commented on Nov 8, 2026

Possibility 2 is closed off: every allocation site in ggml-hrx propagates a failure — so the dropped-"no" hypothesis is wrong

bong-water-water-bong commented on Nov 8, 2026

Binding limit: 9 refusals, no crash, deterministic FAIL — the guardrail's "no" is safe, so the SIGSEGV is somewhere else

bong-water-water-bong commented on Nov 8, 2026

Four repeats on one binary: 2 of 4 crash, always at the same case, and they alternate

bong-water-water-bong commented on Nov 8, 2026

What I tried to reproduce it, and what each attempt showed

The inventory your report needs (all device allocations in the fork, against the pool budget)

Why there is no PR

What would actually close this

bong-water-water-bong commented on Nov 8, 2026

1. Kernel code objects are not the uncharged growth — measured, ~1 MiB

2. What the numbers do point at: the budget is a snapshot, and the ledger is per-process

3. The fix that follows, and why it is yours to choose

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Possibility 2 is closed off: every allocation site in `ggml-hrx` propagates a failure — so the dropped-"no" hypothesis is wrong

Binding limit: 9 refusals, no crash, deterministic `FAIL` — the guardrail's "no" is safe, so the SIGSEGV is somewhere else