[bong-water-water-bong]

third_party/patches/amdxdna: keep the array out of runtime suspend under a live context (#1051)

Two carried amdxdna patches for engine#1051, on top of the series this directory
already ships (0001, 0002, 0004, 0005, 0003).

What the issue is

A dx request that arrives after the NPU's own runtime suspend wedges the array:

npu runtime: invalid call (code 614): state 8, run 0 of 28
aie2_set_cmd_timeout: Context ID: 15 / DPU PC: 0xffffffff / TXN OP ID: 0xffffffff
                      / Timed out sub command ID: 0

The c30/c31 images hide it with the keep-awake rule (one runtime-PM reference
held by the payload while a model is loaded). That is a mitigation in the
caller: every other consumer of the array — a second serve, the demo, anything
that opens a context and then idles 5 s — still hits it.

Root cause, from the driver's own log

Reproduced on strixhalo with the c29 payload + the board's model, power/control=auto,
drm.debug=0x2, 3/3 iterations, on the shipping module (EE1F36A2B9BA294CE11AD80):

Created fw ctx 15 pasid 3
fw ctx 15 host buf op 0x106 addr 0x7f868c000000 size 0x4000000   <- request works
Destroyed fw ctx 15                                              <- autosuspend
Created fw ctx 15 pasid 3
fw ctx 15 host buf op 0x106 addr 0x7f868c000000 size 0x4000000
aie2_hwctx_restart: hwctx.1861315.15 restarted, ret 0             <- driver sees success
   ... first dispatch never completes; TDR ~40 s later

The recreated context gets the same messages, the same firmware context id and
the same host-buffer address as the create that worked, and the restart returns
0. So no driver step that a firmware message can carry is missing; the firmware
accepts the whole restore and then refuses to dispatch into it.

Ruled out on the same box: the resume handshake and the create path (XRT's own
NPU validate — a full context + submit path — passes 5/5 including runs that
start with the device suspended, restarts 0), and the dispatch payload (a
context created after the same resume works). Config cu never appears in
either path because this route has hwctx->cus == NULL, so CU configuration is
not the missing piece.

The fix

0009 — keep the array awake while a context exists (aie2_ctx.c, +35/−0).
A runtime-PM usage reference is taken at the end of a successful
aie2_hwctx_init() (pm_runtime_get_noresume()) and released in
aie2_hwctx_fini() (pm_runtime_put_autosuspend()). Only the usage count
moves: nothing is woken out of order and the create path is unchanged. A device
with no contexts still autosuspends after the existing 5 s delay.

Effect: the array can no longer enter the state whose exit is broken, for any
client, without every client having to know to hold the device awake.

0008 — a dying context must not fail the device resume (aie2_ctx.c +
aie2_pci.c, +33/−4). aie2_hwctx_restart() returns -ESRCH for a context whose
heap address is already gone (the process's mm is torn down before its fds are
closed). That error failed the whole hwctx walk, and aie2_hw_resume() then
broke out of the client loop, so every later client was left with no firmware
context — and pm_runtime_resume_and_get() reported failure for a resume that
had started the hardware, after which amdxdna_pm_resume_get() marks a running
device suspended. That is the board's own line:

amdxdna_pm_resume_get: Resume failed: -3
amdxdna_drm_close: Resume failed (-3), pid 11374 restore data not reclaimed
device is already started

Measured

BEFORE (shipping module EE1F36A2B9BA294CE11AD80, power/control=auto):

• engine ladder, 3 iterations: 3/3 wedge (code 614: state 8, run 0 of 28),
  3 TDRs, log above — ~/lb/k74-ab/i1051-before2/.
• XRT validate across a resume: 5/5 PASS (control) — ~/lb/k74-ab/i1051-xrt-before/.

AFTER (candidate F87D0BDF248067862F83BA3):

• engine ladder (same payload, model and kernels, power/control=auto):
  3 PASS / 0 FAIL / 0 VOID, 0 TDR, 0 amdxdna errors — and the log shows why: the 20 s idle window contains
  no suspend at all (no dpt_suspend, no Destroyed fw ctx in the gap) and no restarted, ret line anywhere in the
  ladder, so the array never entered the state whose exit is broken. ~/lb/k74-ab/i1051-cand-1639/
• XRT validate regression: PASSED (rc 0)
• close-while-suspended A/B, on the 0008-only build F77B98053D0D680105B251F (with 0009 installed the device cannot
  suspend, so that path is unreachable and 0008 needs its own window): arm control=auto clean — the process really
  did exit with runtime_status=suspended and the log shows the suspend (Destroyed fw ctx + dpt_suspend) followed by
  aie2_hwctx_restart: hwctx.798697.15 has no host buffer address (torn down), not restarting it and a completed
  resume (dpt_resume_chan: Resumed), with 0 Resume failed and 0 "restore data not reclaimed". On the shipping
  module the same arm prints the two Resume failed: -3 lines. ~/lb/k74-ab/i1051-close-cand/
• both windows rolled back and verified: loaded EE1F36A2B9BA294CE11AD80, SWAP CLEAN, /boot and the initramfs
  untouched by the test at every step

Limits, stated rather than discovered later

• This does not repair the firmware defect: it removes the trigger. After a
  system suspend (S3), a TDR power-cycle or an admin-initiated suspend, contexts
  still come back through aie2_hwctx_restart() and would still be dead on their
  first dispatch. Fixing that needs firmware/AMD work; 0008 makes the failure
  cost one client instead of every client.
• The NPU stays powered while any context exists. That is the trade the owner
  already accepted at the application level (the c30 keep-awake policy); this
  moves it where it protects all clients, and the array still suspends once the
  last context closes.
• Not measured here: a board (Puppy) run. The box measurement is the shipped
  module vs the candidate module, same payload, same model, same kernels.

Part of #1051.

[bong-water-water-bong]

ready for review at 862dc9a (pi-kernel-74, driver side of #1051). Part of #1051 — cl-1051 leads and its PR carries the Fixes line; this one is the amdxdna half. Measured on strixhalo in two swap windows, each rolled back: 614 ladder 3/3 wedge -> 3/3 PASS (0 TDR), XRT regression PASS, close-while-suspended A/B clean on the 0008-only build (0 Resume failed lines where the shipping module prints them), driver restored to EE1F36A2B9BA294CE11AD80 and SWAP CLEAN both times.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1051 - Partially compliant Compliant requirements: Ensured that a dying context does not fail the device resume Ensured that the array is kept awake while a context exists to prevent runtime suspend Non-compliant requirements: Add a one-line "dispatch at position P / nb N has not completed after X s" (throttled) to make the next board occurrence self-describing Requires further human verification: The specific logging enhancement for dispatch timeouts was not implemented in this PR

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Context Resume Failure The patch modifies aie2_hwctx_resume_cb to handle -ESRCH errors from aie2_hwctx_restart without failing the entire resume operation. This prevents a scenario where a dying context causes the entire device resume to fail, which could leave subsequent contexts in a destroyed state and cause the device to become unusable until a module reload. + ret = aie2_hwctx_restart(xdna, hwctx); + /* + * -ESRCH is aie2_hwctx_restart()'s "this context is being torn down": + * its heap address is gone because the process is closing, so nothing + * will submit on it again and there is nothing to restore for it. + * + * Letting that fail the walk is what took the array away from every + * other client: amdxdna_hwctx_walk() stops at the first error, so the + * contexts of this client after the dying one and of every client after + * it stayed destroyed, and pm_runtime_resume_and_get() then reported + * failure for a resume that had in fact started the hardware. Seen on + * strixhalo as `amdxdna_pm_resume_get: Resume failed: -3` followed by + * "restore data not reclaimed" at a close (engine#1051). + */ + if (ret == -ESRCH) { + XDNA_DBG(xdna, "%s is going away, nothing to resume for it", + hwctx->name); + return 0; + } + + return ret; } Runtime Suspend Prevention The patch ensures that the array is kept awake while a context exists by taking a runtime-PM reference in aie2_hwctx_init and releasing it in aie2_hwctx_fini. This prevents the device from entering runtime suspend under a live context, which was causing dispatch hangs and array wedges. + pm_runtime_get_noresume(xdna->ddev.dev); + XDNA_DBG(xdna, "hwctx %s init completed", hwctx->name); return 0; @@ -886,6 +913,14 @@ void aie2_hwctx_fini(struct amdxdna_hwctx *hwctx) xdna = hwctx->client->xdna; + /* + * Drop the runtime-PM reference aie2_hwctx_init() took to keep the array out of + * suspend while this context existed. Only a successful init takes it (init's + * error paths go to release_expanded_heap/kfree in the create ioctl and never + * reach fini), so this is balanced with it. + */ + pm_runtime_put_autosuspend(xdna->ddev.dev); +

[bong-water-water-bong]

Reproducibility check from origin/main, for a reviewer who does not have my tree

~/lb/k74-i1051-verify-main.sh (task b01524513): a pristine git archive of the pin, then the patch files extracted
from origin/main (not my branch, not my series-base commit) applied in the order the series README documents, with
MODULE_VER_STR='2.21.0_20261104,526c6081974ea4f0f7dd22329186c5d6adb8fad5':

pin: 526c6081974ea4f0f7dd22329186c5d6adb8fad5   (10 files extracted from origin/main @ 4480a6d)
  applied 0001, 0002, 0004, 0005, 0003, 0008, 0009   -- all seven applied cleanly
=== RESULT -> F87D0BDF248067862F83BA3 (want F87D0BDF248067862F83BA3)
    size 14043144  vermagic 7.4.0-rc1-1bit SMP preempt mod_unload modversions
REPRODUCED

So the fingerprint in the README is reproducible from the merged files alone, in the documented order, and the merge did not
mangle a hunk (a mangled patch would fail git apply --check here rather than silently build something else).