npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway

What

The per-op device forward's activation buffer holds exactly max_batch_rows() rows (the M=128 prefill design).
I8Ctx::quantize_async_rows() (npu/forward/engine-src/npu_engine_i8ctx_inc.h:682-703) refuses a larger batch —
prints am=234 exceeds bA capacity MD=128 -- refusing to write past the activation BO — and then returns Am
unquantized, so the caller goes on to launch the kernel and poison/read 234 rows of a 128-row buffer. The result is
memory corruption that is sometimes fatal and always wrong, and the one route that cannot avoid it is a noncausal pack.

Found measuring the d3 route (vllm-sr/d3-lite, engine #796) on a real labelled workload: the published
LocalLLaMA/typed-decisions test split, whose questions are ~225 prompt tokens (usage.input_tokens 1126 for a
5-question case).

Measured

1bit forward-serve -m ~/models/npu/d3-lite-conv/model.q4nx --kernels ~/models/npu/d3-lite-elfs, device path (six
full_i8 designs + six _m16 twins loaded, full ELF ok ×12), one npu-hold ticket per chunk, amdxdna errors 0,
:18097 untouched.

run	requests	result
4 chunks of the labelled split, fresh process each	first request of each chunk	4/4 SIGSEGV: `journalctl -k` → `1bit[3223042]: segfault at 7f3080c00000 ip … in 1bit[38a4c0…]`, and four more 10 s apart, same faulting offset in all four; client sees `RemoteDisconnected` then connection refused
3-case smoke, same binary/container/ELF set, immediately before	3 cases, 15 decisions	answered, but with 1710 `am=234 exceeds bA capacity` lines in the log — i.e. the same overrun, this time landing in mapped memory

So it is not a clean refusal: the same request either dies or answers from rows the kernel never processed. The
"answered" arm's numbers are not usable (rows 128..233 are never passed through the layers), and that is the more
dangerous half.

Why the d3 route cannot dodge it by chunking

forward_serve's chat path already chunks the prefill at max_batch_rows() (app/forward_serve.cpp:325-357,
step_batch(chunk, off, …)). A noncausal_full_attention pack cannot be chunked: step_batch() refuses
start_pos > 0 for it by contract (npu/forward/src/q4nx_forward.cpp:3112-3118), because every full-attention layer
must see the whole prompt in one pass (engine#847). /v1/systemone sends the whole question as one batch, so every
real d3 question is ≥ the 128-row block and lands in this hole.

This also corrects #796's rung 2: the 2026-11-06 device row (10/10 answered) used fixture prompts of ~96 tokens, i.e.
under the block, which is exactly why it passed.

What it would take

Immediate, small: refuse instead of corrupting — step_batch() refuses M > max_batch_rows() on the device path
with a named error (causal callers are told to chunk; a noncausal pack is told it cannot be answered at this length),
so the route answers 500 + reason instead of dying. (quantize_async_rows's early return is already intended as a
refusal; nothing propagates it.)
Capability: blocked noncausal prefill — per layer, fill K/V for every block, then run attention per block, so a
noncausal_full_attention pack can encode a prompt longer than one launch. That is a real piece of work (two passes
per layer and a block-major GDN recurrence), so it wants its own row rather than being smuggled into the refusal.

Repro

~/lb/d3eval/ on strixhalo: fetch.py (the 400 published rows), devrun.sh (one chunk: serve + evalrun.py),
devdriver.sh (four chunks, one ticket each). One request per case, {"state", "questions"} as the pack's contract
requires. Host GEMM (NPU_INFER_HOST_GEMM=1) answers the same requests in 24 s/decision with no device opened, which is
how the labelled row is being taken meanwhile.

How much of the real workload this covers: all of it

Measured with the pack's own tokenizer and its own prompt rendering (d3_runtime.prepare), over the 400 cases of the
published LocalLLaMA/typed-decisions test split (~/lb/d3eval/lens.py):

	tokens
per-question prompt	min 188, p50 376, p95 504, max 687
per-case (5 questions)	min 1088, p50 1889, max 3268
questions over the 128-row activation block	2000 of 2000 (100 %)

So this is not an edge case: with the current kernels a d3 pack cannot answer any published case on the array — every
request either segfaults or answers from rows the layers never processed. The 2026-11-06 device row for engine#796 (10/10)
used fixture prompts of ~96 tokens, i.e. comfortably under the block, which is exactly why it passed and why this went
unnoticed.

Two ways to close it, priced so the choice is explicit (this is an owner decision, not mine to take silently):

Blocked noncausal prefill (the capability): per layer, run the block loop twice — first fill K/V for every block,
then attend per block — so a noncausal_full_attention pack encodes a prompt of any length in one logical pass. Correct
and stays on the device (~3.6 s/decision measured), but it is a real restructure of the batched prefill body (two passes
per layer, block-major GDN recurrence, per-block intermediates held), so it wants its own PR and a device verification
matrix of its own.
Host fallback for over-block prompts: the route keeps a host-GEMM forward beside the device one and answers a
question longer than the block on the host (~24 s/decision measured here, 820 MB extra resident). Small and safe, but it
hides a capability gap behind a latency cliff and mixes two arithmetic paths in one process.

Until one lands, step_batch now refuses by name (PR #1373) instead of corrupting — that is the honest state: d3 on the
NPU answers prompts up to the block; longer questions are refused with the reason and the way to answer them.

Scoping road 1 (blocked noncausal prefill) — it is smaller than it looks, and here is the map

I read the prefill body to price it properly. The noncausal path is already a two-pass-per-layer loop over rows
(npu/forward/src/q4nx_forward.cpp:3403-3411):

if (noncausal_attn_ && l_full) {
    for (int m = 0; m < M; m++) if (!row_attn(m, nullptr, true, 0)) return false;          // pass 1: fill K/V
    for (int m = 0; m < M; m++) if (!row_attn(m, CTX.data()+m*ctx_w, false, start_pos+M))    // pass 2: attend
        return false;
}

What actually breaks is not the row loop but the projection GEMMs over all M rows that sit around it — for each layer,
npu_gemm_rows(i_qkv_l, XN, M, …) before it (~~:3330), npu_gemm_rows(i_o_l, CTX, M, …) after it (~~:3416), then the MLP
GEMMs. max_batch_rows() is the row budget per launch, and every one of those calls hands over the whole prompt.

So the change is: run the layer body once per block of rows, not once per prompt, with the same two passes inside the
full-attention layers.

Block size: the prefill design's M (caps().prefill_rows, 128 today), not max_rows() — for a pb16i container whose
loaded contexts are decode-only, max_rows() is 16 and blocking at 16 would be correct but glacial; the prefill family is
the right budget where it exists.
Per block, the existing sequence is unchanged except the row index: start_pos + m → start_pos + b0 + m, and the
QKV/XN buffers become block-sized (CTX stays M-sized: each block writes its own rows).
Noncausal full-attention layers keep the two passes, per block: pass A for every block first (K/V of all positions
must exist before any row attends), then pass B per block. So the block loop cannot be the outermost loop for those
layers — it is two loops: for b: blocks { pass A } then for b: blocks { pass B }.
Causal layers and the chat path keep one pass per block, which is what forward_serve already chunks at today, so this
makes the forward's own contract match what its callers already assume.
GDN layers: the recurrence is already row-sequential (conv window + recurrent state advance per row), so block order is
the only requirement and a block loop satisfies it. MoE's router/top-k stay per-row.
step() (decode) and step_batch_all() are untouched.

Verification plan (this is what makes it a PR rather than a hope)

row	expectation
host GEMM (`NPU_INFER_HOST_GEMM=1`)	byte-identical answers before/after — the guard and the block loop are device-only, so the reference path must not move
device, prompts ≤ block (the 96-token fixtures)	byte-identical to today's device answers (same launches, same order)
device, the published split's questions (188–687 tokens)	answers instead of SIGSEGV; vs the host-GEMM arm the delta should be the device's int8 grid (~0.09 worst \|Δp\|, per the rung-2 attribution on #796), not 0
the #1372 repro (4 chunks, fresh processes)	0 segfaults, 0 `am=… exceeds bA capacity` lines
`amdxdna` errors, `:18097`, ticket discipline	0 / untouched / one short hold per arm

Estimated: ~150-200 lines of careful edit in one function, plus a device verification matrix of 4 arms; the risk is
concentrated in the two noncausal passes, which is why I would take it as its own PR with the host row as the control
(host is exact and already measured at 24 s/decision).

Still an owner call — road 1 is the real fix and stays on the device; road 2 (host fallback over the block) is ~30 min
but bakes in a latency cliff and a second arithmetic path. Say which and I will run it.

npu: a batch past the activation block is refused by name instead of …

npu: a prefill wider than the activation block is walked in blocks, s…

npu: drop the #1373 refusal now that a wide prefill is walked in bloc…

The dangerous half is confirmed closed on the device, with this issue's own data — and the capability is the next PR

Claiming this row (pi-npufeat): it is this lane's code — I8Ctx::quantize_async_rows's overflow path and the
noncausal contract #847 added. Decision taken here (the issue left road 1 vs road 2 to the owner; the orchestrator
delegates technical calls to the lane): road 1, in two steps. The refusal first (already merged as #1373), then the
capability (blocked noncausal prefill) as its own PR. Road 2 — a host fallback over the block — is declined: it
would add a second arithmetic path with a latency cliff (24 s/decision here) for a pack whose whole point is the array,
and it would hide the missing capability instead of building it.

Verified on the array, current `main` (`6395bbcf`, the #1373 guard), one ticket (14 s wait, 15 s held)

The first published typed-decisions case — the same request that segfaulted 4/4 in this issue — with
1bit forward-serve -m ~/models/npu/d3-lite-conv/model.q4nx --kernels ~/models/npu/d3-lite-elfs (device path, 1 accel
fd, all designs loaded):

HTTP 500  {"detail":"forward step failed: a batch of 234 rows exceeds this geometry's M=128 activation block, and
           attention.mode noncausal_full_attention encodes the whole prompt in one pass, so it cannot be chunked:
           this pack cannot be answered at this prompt length on the NPU (blocked noncausal prefill is missing,
           engine#1372). NPU_INFER_HOST_GEMM=1 answers it"}
serve still alive after the request: yes
"exceeds" lines in the serve log: 0     amdxdna errors: 0

So: no segfault, no am=234 exceeds bA capacity line, no silent answer from rows the layers never processed — a
named 500 that says which limit was hit, that the pack cannot be chunked, and what answers it meanwhile. That is
the property #1373 was merged for, now measured on the workload that found the defect rather than on fixtures.

What is left, and what I will do

The capability: blocked noncausal prefill — per layer, pass A fills K/V for every block, then pass B attends per
block; causal layers and the chat path keep one pass per block (which forward_serve already chunks at
max_batch_rows()). This is the restructure the issue prices at ~150-200 lines in the batched prefill body, and it is
what makes a d3 pack answer the published split on the array at all (100 % of its questions are over the block: p50
376, max 687 tokens). I am taking it as a separate PR with the four-arm matrix this issue already specifies — host-GEMM
byte-identical as the control, ≤128-token device prompts byte-identical to today, real questions answering instead of
500, and the 4-chunk repro at zero segfaults and zero overflow lines.

One correction for the record while I was here: #796's rung 2 (10/10 answered) is still not evidence at real prompt
lengths — those fixtures were ~96 tokens, under the block — so the device row on #1311-style routes stands only for
short prompts until the capability lands. Said plainly because a future reader will otherwise take 10/10 as coverage.

Road 1 is implemented: PR #1382 — and it was 23 lines, not 150-200

I priced two roads above. Road 1 (blocked noncausal prefill) turned out to be engineering rather than a product decision,
so I took it; road 2 (host fallback) is moot now.

My own scoping comment said ~150-200 lines with a two-pass restructure. That was wrong in a useful way: reading the prefill
body properly showed the noncausal two-pass structure was already there and the only over-wide thing was the projection
launches. So the fix is at the one choke point every batched projection goes through — npu_gemm_rows() now walks M rows
in max_batch_rows()-sized launches when a caller asks for more. host_gemm_ returns before the new path, so the
reference arithmetic cannot move.

Measured on the array (six `full_i8` designs + six `_m16` twins, `amdxdna` errors 0, `:18097` untouched)

check	before	after
20 cases / 100 decisions of 188–687-token questions	4/4 SIGSEGV on the first request (fresh process each)	answered, 0 failed, 0 `exceeds bA capacity` lines, 4.0 s/decision
vs the pack's own `d3_runtime.py`, same rows, same q4_1 weights	no number (it crashed)	accuracy 0.4000 vs 0.4000, 0 argmax flips, Brier 0.2628 vs 0.2717, \|Δp\| worst 0.1455 / mean 0.0516
device vs device (same rows, after the rebase that dropped #1373's guard)	—	\|Δp\| worst 0.0000, mean 0.0000, accuracy identical — so dropping the guard is inert and the path is deterministic
host regression, same rows	—	\|Δp\| worst 0.0101, 0 device fds

The 0.1455 is the device's int8 grid (0.1423 measured on fixtures before this change): same argmax, same Brier within
0.009 — the device computes the same thing the reference does, on the hardware grid it has always used.

Disposition

This issue closes with #1382 (Refs #1372, currently MERGEABLE, rebased onto main with #1373's now-superseded
refusal dropped). The row that made it matter — the published LocalLLaMA/typed-decisions split, whose questions are
2000/2000 over the block — is now running on the array in full (400 cases / 2000 decisions, four tickets), so the
next comment on this issue will be that number rather than a crash report.

npu: a prefill wider than the activation block is walked in blocks, s…

Merged as #1382 → 5bdc441b (blocked GEMM in, #1373's superseded refusal removed), so the capability this issue
asked for is on main:

git show origin/main:npu/forward/src/q4nx_forward.cpp | grep -c 'const int blk = max_batch_rows' → 1 (the blocked
path is in), and the M > max_batch_rows() refusal string → 0 (dead code removed with it).
The d3 pack's real questions answer on the array: 20 cases / 100 decisions of 188–687-token questions, 0 failed,
0 exceeds bA capacity lines where the same requests segfaulted 4 times out of 4 before.
Numerically it is the hardware grid, not a different answer: device vs device |Δp| worst 0.0000 on the same rows
(deterministic), and device vs the pack's d3_runtime.py |Δp| worst 0.1455 / mean 0.0516 — the int8 grid that was
measured at 0.1423 on fixtures before this change, with 0 argmax flips on the first 100 decisions and Brier within
0.009.
Labelled, with the benchmark engine#796's rung 3 uses (LocalLLaMA/typed-decisions), first chunk of the full split on
the array: accuracy 0.4100 vs the reference's 0.4320 on the same 500 rows (case-bootstrap CI [−4.4, +0.0] pp),
Brier 0.2388 vs 0.2534, 3.8 s/decision. The remaining three chunks are running so the full 2000-decision device row
exists.

Nothing is left on this issue that a lane can do — the road-2 (host fallback) alternative I priced is moot, and the
blocked prefill is the fix rather than a mitigation. Closing it; the full-split labelled row will land on #796 either way.

npu forward: chunk the pb16i head rows path at the head context's MD,…

Navigation Menu

Repository navigation

npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372

What

Measured

Why the d3 route cannot dodge it by chunking

What it would take

Repro

How much of the real workload this covers: all of it

Scoping road 1 (blocked noncausal prefill) — it is smaller than it looks, and here is the map

Verification plan (this is what makes it a PR rather than a hope)

The dangerous half is confirmed closed on the device, with this issue's own data — and the capability is the next PR

Verified on the array, current `main` (`6395bbcf`, the #1373 guard), one ticket (14 s wait, 15 s held)

What is left, and what I will do

Road 1 is implemented: PR #1382 — and it was 23 lines, not 150-200

Measured on the array (six `full_i8` designs + six `_m16` twins, `amdxdna` errors 0, `:18097` untouched)

Disposition

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: a noncausal (d3) prefill longer than the M=128 activation block corrupts memory — 4/4 segfaults and a 1710-guard-line run that answered anyway #1372

Description

What

Measured

Why the d3 route cannot dodge it by chunking

What it would take

Repro

Activity

bong-water-water-bong commented on Nov 7, 2026

How much of the real workload this covers: all of it

bong-water-water-bong commented on Nov 7, 2026

Scoping road 1 (blocked noncausal prefill) — it is smaller than it looks, and here is the map

Verification plan (this is what makes it a PR rather than a hope)

bong-water-water-bong commented on Nov 8, 2026

The dangerous half is confirmed closed on the device, with this issue's own data — and the capability is the next PR

Verified on the array, current main (6395bbcf, the #1373 guard), one ticket (14 s wait, 15 s held)

What is left, and what I will do

bong-water-water-bong commented on Nov 8, 2026

Road 1 is implemented: PR #1382 — and it was 23 lines, not 150-200

Measured on the array (six full_i8 designs + six _m16 twins, amdxdna errors 0, :18097 untouched)

Disposition

bong-water-water-bong commented on Nov 8, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Verified on the array, current `main` (`6395bbcf`, the #1373 guard), one ticket (14 s wait, 15 s held)

Measured on the array (six `full_i8` designs + six `_m16` twins, `amdxdna` errors 0, `:18097` untouched)