[bong-water-water-bong]

Part of #1051: the long-context half. The runtime-resume half is the driver work in the 0008/0009 patches (the driver-side partner lane), and that PR carries the Fixes #1051 line. This one must not close the issue while that half is open.

What this finds

The 2026-11-04 wedge that #1051 first reported as a long-context hang (614 state 8, run 0 of 28, DPU PC 0xffffffff, prefill position 635, DX_CTX_RESIDENT=0) is not a position-dependent fault, and it is not the resume bug either: the box had power/control=on at the time. It ran on a module without third_party/patches/amdxdna/0003, while other serves held the device. The per-block-count route destroys and recreates its context every 32 positions, so a long prompt goes back to the partition solver ~19 times per 1 200 positions. On that module, each trip could hand it a partition another client already held, which is the two-client wedge measured 3/3 on 2026-11-05. 0003 is already in the series and in the shipping module, so this PR changes docs only: docs/npu.md (Two streams on one NPU) and the 0003 row of the series README.

So the answer to "driver or dispatch" for this half is the driver (the partition solver), and it is already fixed. The dispatch side keeps one documented cost: a client arriving between two block counts of the opt-in per-block-count route takes the free partition, and the first client's request then fails with code 102. It fails with an error, not a hang. The default resident route has no such gap.

Measured proof (strixhalo, 2026-11-06, module EE1F36A2B9BA294CE11AD80, one npu-hold ticket)

The issue's own payload and model (bin 98fb79d5df8a3766, model 6abacb4262504f2a, kernel set 1024x3072x16x8), fresh serve per row, 100 generated tokens per row, NPU held awake, no other /dev/accel/accel0 holder at start. No sudo, no module reload, no power-setting change.

npu-hold -t 15 -l cl1051-longctx2 -- ./longctx-1051.sh     # arms L (ladder), T (two clients), C (client arriving mid-churn)
== #1051 long-context run 2026-11-06T16:29:54-04:00 arms='L T C' ladder='300 640 1200 2150'
preflight: bin=98fb79d5df8a3766 (98fb79d5df8a3766) model=6abacb4262504f2a (6abacb4262504f2a) kernel=7.4.0-rc1-1bit module=EE1F36A2B9BA294CE11AD80
preflight: power/control=on runtime_status=active tdr_timeout_ms= accel0_holders=0
── arm L: single client, ladder, both context modes (100 generated tokens per row)
   L300-pc: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L300-res: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L640-pc: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L640-res: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L1200-pc: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L1200-res: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L2150-pc: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
   L2150-res: http=200 chunks=99 done=True err=''
      tdr=0 errs=0 holders_after=0
── arm T: two clients (the pre-0003 3/3 shape), 1100-token prompt + 100
   Tseq-a: http=200 chunks=99 done=True err=''
   Tseq-b: http=200 chunks=1 done=True err='{"error":{"message":"npu runtime: device error (code 102): No such device"}}'
      tdr=0 errs=3
   Tconc-b: http=200 chunks=1 done=True err='{"error":{"message":"npu runtime: device error (code 102): No such device"}}'
   Tconc-a: http=200 chunks=99 done=True err=''
      tdr=0 errs=3
── arm C: per-count client churning (1100+100), a second client starts 8 s into its request
   C-a: http=200 chunks=1 done=True err='{"error":{"message":"npu runtime: device error (code 102): No such device"}}'
   C-b: http=200 chunks=12 done=True err=''
      tdr=0 errs=3
── smoke after: one short resident row
   smoke: http=200 chunks=12 done=True err=''
== totals since start: tdr=0 amdxdna_errs=9 latched=0 power/control=on 2026-11-06T16:34:50-04:00

Kernel log for the window: 0 TDRs, 0 latches. The 9 amdxdna error lines are the three refusals, 3 lines each (aie2_alloc_resource: Allocate AIE resource failed, ret -19), one per code 102 row.

arm	before (pre-0003 module)	after (EE1F36A2, this run)
one client, per-block-count, ~1 200 positions	1 wedge in 4 rows (2026-11-04, other clients present)	0 TDRs at 300 / 640 / 1 200 / 2 150, both routes, 8/8
two clients, sequential	3/3 wedge, all clients 614 (646FFC30, 2026-11-05)	1st 200 complete, 2nd refused code 102, 0 TDRs
two clients, concurrent	3/3 wedge (same)	1st 200 complete, 2nd refused code 102, 0 TDRs
2nd client arrives during a per-block-count request	(the 2026-11-04 shape)	1st request fails code 102 part-way, 2nd answers 200, 0 TDRs

Smallest context that hangs: none on the current module, in either route. On the pre-0003 module the two-client wedge already hit at the second chunk, so context length was never the variable.

python3 tools/copyright.py --check: rc 0. Docs only, so no build is affected.

🤖 Generated with Claude Code

[bong-water-water-bong]

ready for review at 1ffbfa8

[bong-water-water-bong]

VERDICT: READY at 1ffbfa89 — docs only, the mechanism it names is the patch that is actually in the series, the module id in the text matches this box's own boot evidence, and it correctly does not close #1051. One arithmetic slip in the sentence that carries the argument.

What checks out, all of it read from the tree and the box rather than from the body:

• #1051 is still OPEN — so Part of #1051, no Fixes, is the right form; the body says the driver-side hal...e PR carries the close, and this one must not, which is honoured (gh issue view 1051: OPEN … a dx dispatch can hang once a context is long (614 state 8 …)).
• The patch it credits is the patch that does that. third_party/patches/amdxdna/0003-accel-amdxdna-never-share-an-array-partition.patch — "never share an array partition" is the mechanism the new text describes (a client being handed a partition another client holds), so the sentence "already in the series and in the shipping module" is checkable and true.
• The module id in the prose is this box's own. EE1F36A2B9BA294CE11AD80 in docs/npu.md is the same srcversion the c28 board boot-proof recorded (wt/boardlogs-c28/20261105T234218-9e768484/boot-proof.txt: module.amdxdna loaded=yes srcversion=EE1F36A2B9BA294CE11AD80), i.e. the "fixed" module the measurement claims is the one that actually boots here — not an id typed from memory.
• The README row edit belongs to the same claim (the 0003 row now says what it fixes as well as what it changes), and both files are documentation; nothing in the PR can change behaviour, which is the right shape for "the fix already shipped, here is what it was".
• The run it cites has its artifact on the box: ~/lb/cl1051/longctx-1051.sh (the ./longctx-1051.sh of the quoted transcript).

The nit — one clause, in the sentence the argument rests on: "destroys its context and creates a new one every 32 positions, so a long prompt goes back to the solver ~19 times per 1 200 positions". 1 200 / 32 = 37.5, not 19; 19 is what 1 200 / 64 gives. So either the cadence is 64 positions (and "every 32" is the wrong number), or the count is ~38 (and "~19" is). It does not change the conclusion — a long prompt making dozens of solver trips is the point, and 19 or 38 both mean "many chances to be handed a busy partition" — but the two numbers in one sentence should agree, and this is the kind of pair a reader multiplies first.

Not re-run: the ladder itself (300/640/1 200/2 150 positions × both context routes, 0 TDRs, 8/8 rows) and the two-client arms. That needs the NPU for ~15 minutes plus the issue's payload and kernel set; the c31 gate chain holds device work tonight. I did confirm the preconditions the transcript prints (power/control=on, tdr_timeout_ms=, accel0_holders=0) are the state this box reports, so the measurements were taken the way they are described.

[bong-water-water-bong]

VERDICT: READY at b640399c — main merges only; the reviewed content is byte-identical to the head I verdicted.

Checked rather than assumed: the PR's own added lines (own diff vs merge-base, which excludes everything main brought in) hash to 9c8fe8c306ef at the head I reviewed and to the same value here, so no line of this PR's change moved — only its base did (the branch now sits on newer main).

Everything in my review at the previous head therefore still applies unchanged, including the numbers I re-derived there and the nits I named.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1051 - Partially compliant Compliant requirements: Document the root cause of the long-context hang issue as being related to partition sharing between clients Clarify that the issue was fixed by driver patch 0003 Explain that the per-block-count route can cause a client to fail with error code 102 instead of hanging Document that the resident route does not have this issue and is the default Update documentation to reflect that the driver-side fix is already in place Non-compliant requirements: None Requires further human verification: None

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Documentation Clarity The documentation update explains the root cause of the hang but could be more explicit about the specific error code (102) that clients encounter when partition contention occurs. This would help users understand the failure mode more clearly. - **The "long context" hang was this, too.** The 2026-11-04 wedge that [engine#1051](https://github.com/1bit-MONSTER/engine/issues/1051) first reported as a long-context hang (prefill position 635, the per-block-count route, `power/control=on`) ran while other serves held the device, on a module without `0003`. The per-block-count route destroys its context and creates a new one every 32 positions, so a long prompt goes back to the solver ~19 times per 1 200 positions, and every trip was a chance to be handed a partition another client already held. On the `0003` module (`EE1F36A2B9BA294CE11AD80`, 2026-11-06, one client, the issue's own payload and kernel set, NPU held awake) both routes ran 300 / 640 / 1 200 / 2 150 positions with 0 TDRs, 8/8 rows. Position is not the variable, and there is no position below which it stays safe: the two-client wedges above happened at the second chunk. One cost remains, and it is the per-block-count route's own. Between two block counts that route holds no context, so the partition is free, and a client that arrives in that gap takes it. The first client's next create is then refused (`code 102`) and its request fails part-way (measured, 1/1). It fails with an error, not a hang. The resident route (the default, one context for the process lifetime) keeps its partition and does not have this gap, and the engine's exclusive lock keeps a second serve of ours out of it anyway. Documentation Consistency The README.md update mentions that the fix is reproducible from this directory and references the specific module version (EE1F36A2B9BA294CE11AD80) but could benefit from a clearer statement about the expected behavior after applying the patch (e.g., that clients will fail gracefully with error code 102 instead of hanging). reproducible from this directory rather than from a scratch tree **It is also the fix for the long-context reading of engine#1051** (the 2026-11-04 per-block-count wedge at position 635 ran beside other clients on a pre-0003 module): on `EE1F36A2…` (2026-11-06) one client ran 300-2 150 positions in both context routes with 0 TDRs (8/8), and the pre-0003 two-client shape (3/3 wedges on `646FFC30…`) now refuses the second client `code 102`, 0 TDRs, sequential and concurrent (`docs/npu.md`, *Two streams on one NPU*) |