npu: a dx dispatch can hang once a context is long (614 state 8, DPU PC 0xffffffff), and the TDR then cannot restart a churned context

What happened

On strixhalo (Linux 7.4.0-rc1-1bit, module srcversion 56AB94758C18FEEC838D18C, tdr_timeout_ms=15000 from
/etc/modprobe.d/amdxdna-tdr.conf), with the c24 payload binary (sha256 98fb79d5df8a3766), the board's own model
(6abacb4262504f2a, Qwen3-0.6B) and the board's kernel set (1024x3072x16x8), this is the first reproduction on the box
of the failure the c22/c24 Puppy boards showed:

{"message": "npu runtime: invalid call (code 614): state 8, run 0 of 28"}

with a real TDR in the same second:

aie2_legacy_tdr_detect: TDR timeout detected
aie2_tdr_work_func: Device isn't making progress
aie2_set_cmd_timeout: Context ID: 15 / DPU PC: 0xffffffff / TXN OP ID: 0xffffffff
                      / Context PC: 0x28b060ad / Timed out sub command ID: 0
aie2_tdr_recover_all: Array stopped dispatching, power-cycling NPU to recover
aie2_hw_reset: NPU firmware unhealthy: power-cycling NPU
aie2_hwctx_restart: hwctx.<pid>.15 has no host buffer address, cannot restart it
aie2_tdr_resume_hwctx: hwctx.<pid>.15 restart failed, ret -19
aie2_tdr_recover_all: NPU array is unusable until the driver is reloaded: TDR recovery failed
                      (0, a context could not be restarted); new contexts and submissions are refused with -ENODEV

The engine side of it, per token (DX_PROFILE=1), is a discrete hang, not a growing step: 634 prefill dispatches at
layers 12-13 ms, ctx <= 0.001 ms, then a dispatch whose rl.execute()/rl.wait() never returns (the last profile row
is ctx runlist S with no layers). The hang is in prefill at prompt token ~635 (attention block count 20, i.e. 25
tokens after the last block-count switch).

Reproducer

An 1100-token prompt + 100 generated tokens, board payload + board model + board kernel set, fresh serve per row, under
an npu-hold ticket. Harness and per-token tables in the shared evidence dir:
~/amd-oss/hrx-study/evidence/fabric/npu-longctx/engine-20261104-214924/ (README-engine.md, serve-A.log,
engine-dmesg-A-B.txt, prompts/) and engine-rate-*/.

It is not deterministic: the identical re-run after a module reload, and every later row of the same prompt in either
mode, passed (1200 positions, 19-24 s each, 0 TDR). Rate in the same 1100-token-prompt shape (engine-rate-20261104-215354/): 1 of 4 per-count rows wedged (the first
one), 0 of 3 resident rows in the same window — and the wedge happened at load1 76 while every clean row ran at
load1 33-37. The driver lane's ~15 resident rows tonight were clean too. So the box is a low-rate reproducer while
the boards (also resident) wedge far more often: the variable left is the host (Puppy: OS in RAM, no swap, zram,
Lemonade in front), not the sequence position.

The one wedge so far was in DX_CTX_RESIDENT=0 (per-block-count contexts: the hw_context is rebuilt at every block
count, and this is Impl::resident_context()'s own fallback when add_config() throws). Whether the resident path — the
default, and what the appliance runs — can hang too is not established: the rows of it measured so far are clean.

Why this is worth an issue even though it is a private add-on path

The hang itself is a firmware stall (DPU PC 0xffffffff, "Timed out sub command ID: 0", run 0 of 28 — chain 0's
first sub-command; the runtime submits 28 runs as 2 ERT_CMD_CHAIN packets because kChainMax = 24), and it is the
same signature the two boards produced on their second/third long request. Nothing about it is position-specific:
the position differed between the boards (nb 20 on c22, nb 36 on c24) and differed again here, which is what a
probabilistic hang looks like when each ladder happens to be standing at a different place.
The consequence is worse than the hang: aie2_hwctx_restart cannot restart the churned context ("has no host buffer
address"), so the whole box loses the NPU (-ENODEV, every new context refused) until amdxdna is reloaded. A
single stuck dispatch should not take the device away from every other engine.
The engine's own contribution is avoidable: the resident path (one hw_context for the process lifetime, a new block
count is one add_config()) is the default and was measured clean at 1511/2146/3408 positions tonight; the
per-block-count churn exists only as an optimisation fallback. If add_config() fails, failing the request is better
than silently degrading into a mode that can brick the array.

Asks

add-on owner: (a) the per-count fallback is a liability on top of the hang — if add_config() is refused, fail the
request instead of churning contexts, so a wedge cannot also destroy the TDR recovery; (b) the engine waits on the
runlist with an infinite timeout, so a hung dispatch has no trace of its own position — a one-line "dispatch at
position P / nb N has not completed after X s" (throttled) would make the next board occurrence self-describing;
driver: aie2_hwctx_restart should not fail on a context the driver created — or the TDR recovery should destroy and
recreate instead of restarting, so a single hang costs one request and not the array.

Board context and the ladder that produced the c22/c24 rows: ~/briefs/TC22-NPU-TDR.md. Engine-side measurements and the
per-token tables: the evidence dir above.

Engine patch written, built and device-checked (the two asks above)

addons/npu-dx/src/dx_decoder.cpp, +17/-4 (private add-on, so a patch rather than a PR):
~/briefs/0001-npu-dx-stall-watch-and-no-per-count-fallback.patch, mirrored in the evidence dir as
engine-1051-stall-watch-diff.patch.

A hung dispatch names itself. step() now waits on the runlist in 5 s slices instead of one infinite wait and
prints, per slice,
dx: dispatch at position P (N attention blocks) has not completed after X s.
A slice timeout is not a failure: the sequence may still signal and a non-completed packet still throws 614 from the
final wait, unchanged. Nothing is added to the fast path — the first wait returns as soon as the run completes.
No silent per-block-count fallback. resident_context() throws
dx: the resident context refused block count N (why); this kernel set cannot serve that position in one context
instead of rebuilding the hw_context every 32 positions. That path is the only engine structure that churns contexts,
and a hang inside it is what turned the wedge into a device-wide outage on strixhalo
(aie2_hwctx_restart: ... has no host buffer address → -ENODEV → modprobe -r amdxdna). Tradeoff, stated in the
patch: a kernel set that legitimately cannot add a config above some block count will now fail the request instead of
degrading — no such refusal has been observed up to block count 107 on this box.

Built (-DONEBIT_NPU=ON, this add-on, engine tree from this lane) and run on the device in the board's configuration
(board kernel set 1024x3072x16x8, Qwen3-0.6B, ctx 32768, 1100-token prompt + 100 tokens, streamed, fresh serve per row,
npu-hold ticket):

row	mode	result
res-1..4	resident	200, 20.1 / 16.1 / 15.1 / 17.2 s, 100 SSE lines, 0 stall lines
pc-5	`DX_CTX_RESIDENT=0`	200, 23.1 s, 100 SSE lines, 0 stall lines

TDR count and amdxdna error count were unchanged across all five rows (the single TDR since boot is the wedge this
issue is about). No row wedged in this window, so the stall line itself is not yet exercised on hardware — it is a plain
loop over the runlist wait's timeout, and it will print the position the first time a dispatch hangs.

Evidence, per-token tables and harnesses: ~/amd-oss/hrx-study/evidence/fabric/npu-longctx/
(engine-20261104-214924/ for the wedge, engine-rate-20261104-215354/ for the 1/4 vs 0/3 rate,
engine-patch-20261104-220052/ for these rows).

Traceability, now that the owner has taken the decision: the add-on change is in the private repo — 1bit-MONSTER/npu-kernels PR #26, branch 1bit/npu-1051-stall-watch, commit 9c589b47, based on origin/npu-dx (59ee2fdc). It carries the two hunks described above: the 5 s-slice wait that names a hung dispatch (dx: dispatch at position P (N attention blocks) has not completed after X s) and resident_context() throwing instead of falling back to one context per block count. The patch file is also in the evidence dir and in ~/briefs/0001-npu-dx-stall-watch-and-no-per-count-fallback.patch, so it can be re-applied if the branch is dropped.

Two device facts from tonight that belong in this issue rather than only in the state file:

6 rows of ~1200 positions on driver 0008 were clean (3 resident + 3 per-block-count, board payload, 0 TDR, amdxdna error count unchanged) — the opposite of the 1-in-4 per-count rate measured on 0007. Six rows cannot separate a fix from a low base rate, so this is not a claim that 0008 fixed the hang; it is the first 0008-era data point and it is in ~/amd-oss/hrx-study/evidence/fabric/npu-longctx/after-0008-20261104-235035/.
The consequence is still live: the failed aie2_hwctx_restart path (has no host buffer address) leaves the array refusing every context until amdxdna is reloaded, and on this box that reload is additionally blocked while any HRX llama-server holds /dev/accel/accel0 — an unrelated GPU backend pinning the NPU node is what stopped two in-ticket recoveries tonight. That one is for the driver/HRX owners.

Correcting my own numbers upward with the second ticket: nine consecutive clean per-block-count rows of ~1200 positions on driver 0008 (3 in after-0008-20261104-235035/, 6 in engine-614row-20261105-000052/), all 200 in 16-21 s, no stall line, module 6483D317… unchanged across both, Array stopped dispatching and amdxdna.*ERROR unchanged (12 / 444). Under the 0007 rate of 1 in 4, nine clean rows would be a ~7.5 % coincidence — suggestive that 0008 changed the behaviour, and explicitly not proof that the hang is fixed: my one 0007 wedge ran at load1 76 while every clean row ran at 33-37, so a load-dependent hang would not appear in a quiet window.

Consequence: the 614 row (this issue's sibling, #1021 / TC22 item 2) is still unmeasured — no wedge to measure it on. The harness that would record it is ready and documented (harness/engine-614row.sh): on a 614 it stops killing the serve, records whether the process leaves by itself with exit code and delay, then whether a fresh backend answers, then reloads in-ticket if the driver left the array -ENODEV.

serve: one line per request and one per Laya decision, with the value…

Engine side of the concurrency trigger, for the record on this issue (it does not fix the long-dispatch stall
itself).

The in-process route (1bit serve -m <q4nx model dir> --device npu, app/unified.cpp — the route the payload's
Lemonade backend takes) took no cross-process device lock, while forward-serve has taken one since engine#612.
So the hwctx_num = 2 shape of the 03:46 reproduction was two of our backends, not a foreign runtime.
engine#1094 makes that route take the exclusive lock before it opens the model; two serves now serialise
(Pair row: A ready in 610 ms, B waits; counters 2410 → 2410 amdxdna ERROR, 0 → 0 TDR).

What that leaves on this issue: a single long dx dispatch can still hang (#1051's own reproduction was one
client), and the driver still allows a foreign context to share the partition (#1084). Both halves are yours;
this one only removes the trigger the engine was contributing.

The trigger is the device's own runtime-resume, not context length. Reproduced 4/4 on the c29 payload binary
(sha256 89a6c386…) + the board's model snapshot (b5847992…) + the same dx kernels, on an identical
amdxdna module (0.18.0, srcversion EE1F36A2B9BA294CE11AD80) — and 7/7 clean runs when the device is not
allowed to suspend:

arm (fresh process, board shapes: game turn then picture prompt)	device may autosuspend	result
two requests back to back	yes	pass
load→close, then two requests	yes	2/2 pass
two requests, 20 s idle between them	no (`power/control=on`)	pass
two requests, 20 s idle between them	yes (`control=auto`)	4/4 fail: `code 614: state 8, run 0 of 28` after 35–45 s, 14 `amdxdna` errors, TDR, `DPU PC 0xffffffff`, `TXN OP ID 0xffffffff`, `Context PC 0x28b060ad` — byte-identical report to the c29 board take

The idle window is what matters because AMDXDNA_AUTOSUSPEND_DELAY is 5 s (amdxdna_pm.c:13): the engine's next
request needs a resume, and its first dispatch into the context that resume recreated never completes.

Why this box had never shown it: this host ships /sys/bus/pci/devices/0000:c6:00.1/power/control=on, so the
NPU never runtime-suspends here. echo auto makes the c29 fault deterministic (4/4); echo on makes it vanish.
That also explains the earlier "long prompt" reading — a long prompt is simply the request that usually arrives
after a gap.

The c29 take log anchors agree with this, once corrected: lemond stopped the first NPU backend at 22:14:58.4
and the kernel logged amdxdna_drm_close … pid 11374 restore data not reclaimed at uptime 141.478 s ⇒ that boot
began 22:12:37 (the release tag's 22:13:28 is the boot-proof snapshot, the same 51 s trap as c28). With that
anchor the TDR at 304.161 s is 22:17:41.2 — the same 0.1 s as the engine's code 614 line, i.e. the TDR and
the 614 are one instant, not 50 s apart. That close (loaded, idle, suspended context) hit the same defect from
the other side: amdxdna_pm_resume_get: Resume failed: -3 → restore data left behind.

Reproducer (any box, same payload/model/module): ~/wt/npufeat-614/arm614.py W under npu-hold, wrapped by
repro614w.sh, which flips power/control to auto for the run and back to on before the ticket is released.

The driver question that stays open: aie2_hwctx_stop() destroys the firmware context on suspend and
aie2_hwctx_restart() recreates it on resume (create_context + map_host_buf + the expanded heaps up to
last_attached_heap + config_cu, CU BOs held per 5d268cc8). The first dispatch into that recreated context
never reaches the DPU. Nothing in the driver log reports a failure on the resume path — it looks successful and
the array is wedged on the next submit. That is yours (pi-kernel-74); I have the trigger and a reproducer that
does not depend on request shape.

Engine side (this lane): a serve that holds the array now keeps one runtime-PM reference alive while a model is
loaded (a GET_INFO every 3 s — the same amdxdna_pm_resume_get()/suspend_put() pair the submit path takes),
so the resume is never entered. PR to follow with the 0-failure rows.

Disposition: root cause identified; mitigated by an owner decision; the driver-side fix remains. Parked here with that split rather than closed.

Trigger, measured 4/4 vs 7/7 in the thread: the device's own runtime-resume (AMDXDNA_AUTOSUSPEND_DELAY = 5 s), not context length. The first dispatch into the context a resume recreated never completes; control=on makes the failure vanish.
Owner decision 2026-11-05 23:4x ("keep the npu awake on the stick") is the shipped mitigation: the image sets power/control=on for amdxdna (with an early-boot fallback), and the release check asserts the NPU is held awake — that is PR puppy: the NPU never runtime-suspends on the stick (the c30 614 fix) #1230 (image half, based on puppy: hosts.allow decides by address (no name test), and lemond stops logging an offline update check #1224) and PR release-check: assert the NPU is held awake (power/control=on, runtime_status=active) #1231 (release-check half) on the c30 line.
The driver-side fix (a resumed context whose first dispatch hangs) is amdxdna/kernel work, not engine work; the TDR-recovery series (0008/0009(+0010), owner "go A") is a different failure and is installed separately.

Unpark: the driver fix lands (or the keep-awake policy is dropped); then the c30 614 rows are the acceptance (the take's demo sequence 10x, PR #1225's probe).

Sweep (pi-verify) at `918f731a` — still open, not re-run in the sweep

The reproducer is an NPU ticket (1100-token prompt + 100 generated tokens, fresh serve per row), and the box's NPU queue is
occupied by another lane's long-context row, so the sweep did not re-run it. What I can say about current main is limited on
purpose: the dispatch path that wedges is not in the public tree, so I could not read whether the chain submission changed — the
only public-side change in this area is the device lock (ad472533), which prevents two backends dispatching at once and does not
address a firmware stall. The low-rate reproducer stands as filed.

third_party: the amdxdna pair for #1051 (no runtime suspend under a l…

docs/npu: the long-context reading of #1051 is the partition share 00…

Closing: both halves are fixed in the series and measured before/after (#1294, #1295)

#1294 and #1295 both said "Part of #1051", each leaving the Fixes line to the other, so nothing closed this. Here is the evidence for each part of the title, all from strixhalo on 2026-11-06.

1. The long-context hang (614 state 8, DPU PC 0xffffffff, position 635) — #1294. This is not a position-dependent fault. On a module without 0003, a client that goes back to the solver can be handed a partition another client already holds. The per-block-count route does that every 32 positions, and other serves held the device at the time. On the shipping module EE1F36A2B9BA294CE11AD80 (which carries 0003), with the issue's own payload, model and kernel set and the NPU held awake:

one client, 300 / 640 / 1 200 / 2 150 positions, per-block-count and resident: 8/8 complete, 0 TDR;
the pre-0003 two-client shape (3/3 wedges on 646FFC30 on 2026-11-05), run sequentially and concurrently: the second client is refused with code 102, the first completes, 0 TDR.

2. The resume-path hang (first dispatch into a context recreated after runtime suspend) — #1295, 0008 + 0009.

before, shipping EE1F36A2, power/control=auto: 3/3 wedge (code 614: state 8, run 0 of 28, 3 TDRs);
after, F87D0BDF248067862F83BA3 (series + 0008 + 0009), same payload, model and kernels: 3/3 PASS, 0 TDR, 0 amdxdna errors. There is no suspend in the idle gap and no restarted, ret line, and the XRT validate regression passed;
0008 alone (F77B98053D0D680105B251F), close-while-suspended A/B: shipping prints Resume failed: -3 / "restore data not reclaimed"; the candidate is clean.

The merged files are the measured ones. third_party/patches/amdxdna/0008-*.patch and 0009-*.patch at af36665b are byte-identical (cmp) to the files the candidate was built from.

3. "The TDR then cannot restart a churned context" (has no host buffer address → -ENODEV latch): 0004 (a closing context is skipped, not a failure) and 0005 (throttle checked before touching any context), shipping since c26. Today's run had 0 latches, including three code 102 refusals.

What stays open, outside this issue:

Installing 0008/0009 is the owner's call. The box and the c30/c31 images still run EE1F36A2, where the keep-awake rule (puppy: the NPU never runtime-suspends on the stick (the c30 614 fix) #1230/release-check: assert the NPU is held awake (power/control=on, runtime_status=active) #1231) is the mitigation. A signed F87D0BDF build and a rollback copy exist (see the third_party/amdxdna: no runtime suspend under a live context, and a dying context must not fail the resume (#1051) #1295 body).
The firmware defect itself: a context recreated after a forced suspend (S3, TDR power-cycle, admin) is still dead on its first dispatch. 0009 removes the runtime-suspend trigger and 0008 limits the cost to one client; repairing the firmware needs AMD. third_party/amdxdna: no runtime suspend under a live context, and a dying context must not fail the resume (#1051) #1295 states this limit.

The frozen c32 image carries the right driver — checked from the image bytes, not from the build log

~/wt/release-rc/frozen/rc-2026-11-07-c32-6456d7bb34cf.img, read-only, nothing installed or changed:

what	where	value
`amdxdna.ko` in the zdrv	`zdrv_upup64r_26.04.sfs` → `usr/lib/modules/7.4.0-rc1-1bit/updates/amdxdna.ko`	srcversion `F87D0BDF248067862F83BA3`, sha256 `b76fb30d303a98ad…`, 14 043 025 B, signer `strixhalo Secure Boot Kernel`
the same file the swap windows measured	—	identical sha256 → the image ships the artifact that was measured, not a rebuild that merely has the same name
`ec_su_axb35.ko`	same `updates/`	present, 414 217 B, signer `strixhalo Secure Boot Kernel`
the initrd	`1bit/initrd.gz`	carries no kernel modules at all (`find . -name '*.ko'` = 0, no `lib/modules`), so the stick has no second copy to go stale — the initrd-shadowing trap that bit the Ubuntu install on 2026-11-04 does not exist on this layout
the NPU keep-awake rule	`puppy_upup64r_26.04.sfs` → `etc/udev/rules.d/90-1bit-npu-power.rules`	present (2448 B) and redundant-but-harmless now that 0009 keeps a live context awake from the driver side
the gate's expectation	`wt/release-rc/puppy-check.sh`	`AMDXDNA_SRCVERSION=F87D0BDF…` with `EE1F36A2…` also in `AMDXDNA_KNOWN` — so the row reds on a stale module and still accepts a previous cycle's artefact by design

My 91-1bit-xhci-keepawake.rules (#823, draft) is correctly absent from c32 — it should not ship before a drive has
actually been plugged in.

Nothing here needed a device window: the image is a file, and every row above is read out of it.

docs/npu: the amdxdna series is nine patches, seven installed — say w…

Navigation Menu

Repository navigation

npu: a dx dispatch can hang once a context is long (614 state 8, DPU PC 0xffffffff), and the TDR then cannot restart a churned context #1051

What happened

Reproducer

Why this is worth an issue even though it is a private add-on path

Asks

Engine patch written, built and device-checked (the two asks above)

Sweep (pi-verify) at `918f731a` — still open, not re-run in the sweep

Closing: both halves are fixed in the series and measured before/after (#1294, #1295)

The frozen c32 image carries the right driver — checked from the image bytes, not from the build log

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

npu: a dx dispatch can hang once a context is long (614 state 8, DPU PC 0xffffffff), and the TDR then cannot restart a churned context #1051

Description

What happened

Reproducer

Why this is worth an issue even though it is a private add-on path

Asks

Activity

bong-water-water-bong commented on Nov 5, 2026

Engine patch written, built and device-checked (the two asks above)

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 5, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

Sweep (pi-verify) at 918f731a — still open, not re-run in the sweep

bong-water-water-bong commented on Nov 7, 2026

Closing: both halves are fixed in the series and measured before/after (#1294, #1295)

bong-water-water-bong commented on Nov 8, 2026

The frozen c32 image carries the right driver — checked from the image bytes, not from the build log

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

Sweep (pi-verify) at `918f731a` — still open, not re-run in the sweep