hrx: test-backend-ops fails 324/4374 cases deterministically — all 320 TOPK_MOE, 4 of 6 DSV4_HC_POST

test-backend-ops test -b HRX0 is not green on this pin, and the failures are the same every run. Gates and PRs that cite "the HRX suite passes" should cite this issue until it is fixed or the cases are waived on purpose.

What was run

Binary: the fork at cffae5c4 (HRX build, gfx1151), on strixhalo, 2026-11-06 ~03:37 AST.
Command: ./test-backend-ops test -b HRX0 (the full suite, not filtered with -o), run twice back to back.
Result: 4050/4374 passed both times, 324 cases failed. The same 324 case names fail in both runs; only the last digits of some ERR values change.
Evidence on strixhalo: ~/lb/aval/tattn-jit-scope-20261106T063616Z/full-1.log, full-2.log, inner.sh.

The failures

1. TOPK_MOE: 320 of 320 cases fail, every shape and every gating_func (0-3), with and without with_norm, bias_probs, scale_w.

Almost every case reports [VIEW] ERR ≈ 1.0. Some also report [RESHAPE] or [SCALE] errors.
The test sorts both outputs before comparing them (test_topk_moe::err → nmse). So this is a value mismatch, not a tie-order difference in the expert ids.
HRX does claim at least part of this subgraph: the log shows ggml_hrx dispatch: llm.moe_router.top8_f32 (fused). The failures are therefore not only the harness splitting the graph around an unclaimed node.
Two cases (ne=[31,22,1,1], n_expert_used=8, with_norm=1, with scale_w 0 and 2) fail earlier, at kernel compile time:
```
HRX Loom JIT ... target 'amdgpu-rdna3-5' export 'qwen3_moe_router_top8_f32' ... rejected 'index.assume' ...
predicate 'range' on 'experts_per_lane0' requires range [1, 16] but ...
graph_compute: failed to prepare command 0 kind=Kernel ...
```
With 31 experts the matcher accepts a shape that the kernel's range assumption does not allow. The matcher should refuse that shape, or the assumption should be widened.

2. DSV4_HC_POST: 4 of 6 cases fail. All four are identity=0; both identity=1 cases pass.

case	ERR (bound 1e-7)
`n_embd=1, n_tokens=1, identity=0`	0.078
`n_embd=31, n_tokens=17, identity=0`	0.217
`n_embd=128, n_tokens=257, identity=0`	0.174
`n_embd=4096, n_tokens=21, identity=0`	0.174

The node goes to hip.hc_post (single), which is an HRX dispatch. These look like a real kernel gap on the non-identity path.

Why this matters

The open question is whether real MoE decode ever reaches the failing router path. Our measured MoE runs (Qwen3-30B-A3B and others, judged on KLD) look correct. So either the fused router is not selected for those models' shapes, or the suite tests a layout that real graphs do not produce. Neither is shown yet.
Gates that run the suite with -o <op> filters never see these cases. The overnight TOP_K investigation in TATTN ran 180 filtered runs with no failures, and none of them covered this.

Asks

Re-run the full suite at the current engine pin (third_party/llama.cpp at 710c7c23) under gpu-hold. Post whether the counts change.
TOPK_MOE: find which node the [VIEW] mismatch is on. Then decide whether it is a kernel bug or a harness or graph-shape artefact, and state whether a real model hits the fused router.
Make the qwen3_moe_router_top8_f32 matcher refuse expert counts outside the kernel's assumed range (the 31-expert compile failure).
DSV4_HC_POST: fix the identity=0 path of hip.hc_post, with the four cases above as the test.
Until 1-4 are done, a gate that reports the HRX suite reports 4050/4374, not "green".

Found by pi-attn (TATTN, 2026-11-06 03:41 AST); counts re-checked from the logs by the orchestrator.

ask 1 — the pin does not build, and after the one-line build fix the counts do not change

The pin could not be built at all

third_party/llama.cpp at 710c7c23 = tip of 1bit/hrx-vulkan-patched (0 commits ahead). ninja test-backend-ops stops at
llama-context.cpp:

src/llama-context.cpp:160:42: error: cannot pass non-trivial object of type 'std::string' (aka 'basic_string<char>')
to variadic function; expected type from format string was 'char *' [-Wnon-pod-varargs]
FAILED: src/CMakeFiles/llama.dir/llama-context.cpp.o

Cause: f1487819e (engine #681, merged in #219) passes model.arch_name() — std::string per src/llama-model.h:742 — to
LLAMA_LOG_WARN's printf-style %s. Nothing on the branch tip compiles from source. Fix is one line
(arch_name().c_str()), PR llama.cpp #220, head bd0762ca3, base 1bit/hrx-vulkan-patched, MERGEABLE, pi-prs pinged.
The run below is that pin plus that one line; nothing else differs.

Counts at the pin (with the build fix), `test-backend-ops test -b HRX0`, twice back to back

	old build (cffae5c4)	pin 710c7c23
passed	4050 / 4374	4275 / 4599
failing cases	324	324
identical failing set between the two runs	yes	yes

The totals moved because the pin carries 225 more cases (15 commits' worth of new kernels/tests), not because anything was
fixed: the failure structure is unchanged — [VIEW] 320 (all 320 TOPK_MOE cases) + DSV4_HC_POST 4, byte-identical between
the two runs. Evidence: ~/lb/aval/tattn-1259-20261106T133039Z/full-{1,2}.log.

ask 2 — which node the `[VIEW]` mismatch is on, and whether a real model reaches the fused router

The node

[VIEW] is not a tensor name: the harness prints ggml_op_desc(t) (tests/test-backend-ops.cpp:1494), i.e. the op, and
ggml_argsort_top_k (ggml.c:5365) is argsort(DESC) + ggml_view_4d(..., k, ...). test_topk_moe::fusion_test_nodes()
returns exactly {selected_experts, weights}, so the two compared tensors are the id tensor (op = VIEW) and the gathered
weights. Only the ids mismatch; the weights node passes. Which matters, because the weights are gathered from those
ids: for distinct probs a different id set means a different weight multiset, so weights passing means either the two id sets
differ only inside a tie group, or the weights comparison is not seeing the difference.

Which dispatch produces those ids (from the dispatch log, not from reading)

For shapes the fused router does not take, the chain runs on HRX's small-rows family:
common.argsort_rows_f32 + common.get_rows_small_f32 + common.sum_rows_f32 (all three appear in a full-suite dispatch
log). match_argsort_rows accepts any F32→I32 argsort with ne[0] <= 1024 (dispatch-small-rows.cpp:130-150).
For eligible shapes the fused router fires and covers the id node itself: append_moe_router_top8_coverage lists
topk_view among the ten nodes it claims, so when llm.moe_router.top8_f32 matches, the ids come out of
qwen3_moe_router_top8_f32.
Measured per shape (filtered runs, GGML_HRX_LOG_DISPATCH=1): ne=[32,22], n_expert_used=8 → fused (moe_router
logged 7x); ne=[8,22] k=4, ne=[31,22] k=8, ne=[128,1] k=128 → not fused (ids from argsort_rows_f32).

Do real models reach the fused router? Yes — measured, three logs

GGML_HRX_LOG_DISPATCH=1 server logs on this box show ggml_hrx dispatch: llm.moe_router.top8_f32 (fused) at load for:

Qwen3.8-Flash-Next UD-IQ3_XXS — the hrx: long-context decode 9.70 tok/s at 18.8k ctx (prefill 74-93) vs 31.15 / 459 on a gfx1151 fork — same GGUF, same hour, short-prompt parity 28.6 vs 28.9 #679 model — in two separate serves
(~/lb/aval/tattn-rx-reach/20261104T062343Z/server.log, ~/lb/aval/tattn-18k-window-floor-20261105T092045Z/server.log);
Qwen3.6-35B-A3B UD-IQ4_XS (~/lb/aval/tattn-qwen36-decode-20261103T223743Z/qwen36-c131072.server.log).

Matcher bounds agree with that: is_supported_expert_count = [32, 512] and % 32 == 0, is_supported_route_count = 1..32
(dispatch-moe-router.cpp:95-101) — so Qwen3-30B-A3B (128/top-8) and Qwen3.6-35B-A3B qualify, and most of the suite's shapes
(8, 31, 40, 71, 129) cannot be fused at all. So the fused router is on the hot path for our measured MoE models — which is
consistent with those runs being KLD-correct, and means "the fused router is wrong" cannot be the whole story.

The new datum that changes the question

The same cases pass when the suite is filtered to them and fail in the full run. Filtered runs of four of the failing
cases at the pin ([128,1] k=128, [32,22] k=8, [8,22] k=4, [31,22] k=8) all returned rc=0; those same cases are among
the 320 that fail in the full suite. So this is not a pure shape/kernel bug: it depends on the process state built up by the
preceding ~4000 cases. That is the same class as the TOP_K flakiness I closed out yesterday (identical binary, dirty in one
claim and clean in another), and it is why I am not calling it a kernel bug yet.

Next (queued): one full-suite run with a dump that prints only mismatching comparisons — both sides plus sorted copies of
the ids — so the ids the suite actually reads are on the record instead of inferred. Will follow up here with that.

One latent matcher bug found on the way (same class as ask 3)

match_argsort_rows accepts ne[0] <= 1024, but ggml_argsort_rows_f32 assumes range(%column_count, 1, 1024) and
range(%rank, 0, 1023) (small_rows_f32.loom), i.e. its own bound is < 1024. ne[0] == 1024 is therefore accepted by the
matcher and outside the kernel's assumption — exactly the shape of the 31-expert qwen3_moe_router_top8_f32 reject in the
issue body. No TOPK_MOE case in the suite has 1024 experts, so it is not tonight's failure, but the same fix pattern applies.

ask 2 (final) — the `[VIEW]` mismatch is the expert-id tensor coming back un-written, and the producer is `common.argsort_rows_f32`

Follow-up to my previous comment. Numbers below: pin 710c7c23 + the one-line build fix (PR llama.cpp #220), full
test-backend-ops test -b HRX0, one claim, three arms
(~/lb/aval/tattn-1259c-20261106T140044Z/).

arm	passed	`[VIEW]` fails	TOPK_MOE fails	DSV4_HC_POST
control	4275/4599	320	318	4
`GGML_HRX_DISABLE_DISPATCH=common.argsort_rows_f32`	4229/4273	40	6	4
`GGML_HRX_DISABLE_DISPATCH=llm.moe_router.top8_f32`	4275/4599	320	320	4

1. What the failing tensor actually contains

A dump inside a full run (~/lb/aval/tattn-1259b-20261106T134148Z/full.log, 640 mismatch lines, dump patched to print only
mismatching comparisons) shows, for the ids tensor:

[#1259 MISMATCH n=88 ndiff=88] HRX: -1.08297e+09 1.05857e+09 -1.08762e+09 ... | CPU: 0 6 7 2 5 6 3 2 0 3 5 6 ...

The CPU side is real ids. The HRX side is ±~1.08e9 int32, which are the float bit patterns of the argsort input
(0.5f = 1056964608, -1.0f = -1082130432). So the ids output is not written on the HRX path — the buffer still holds
the data the previous kernel put there. The gathered weights node mismatches as well (ndiff 42/88), so this is not the
"sorted comparison hides an order difference" artefact I first hypothesised: the ids are genuinely absent.

2. Which dispatch

common.argsort_rows_f32 is the producer: disabling that one matcher drops [VIEW] failures 320 → 40 and
TOPK_MOE failures 318 → 6. That kernel (kernel-corpus/kernels/loom-libs/ops/small_rows_f32.loom) computes each
element's rank and does one store, guarded by a data-dependent assume:
```
%rank_bounded = index.assume %rank [range(%rank, 0, 1023), lt(%rank, %cols)] : index
view.store %element_i32, %output_view[%row, %rank_bounded]
```
%rank is computed at run time, so lt(%rank, %cols) is not something the compiler can prove — that assume is the only
guard around the single store, and it is the first thing I would look at (an assume that the JIT turns into a predicate
would silently drop the store, which is exactly the "ids hold the previous kernel's bytes" signature).
The fused router is not the primary suspect: disabling llm.moe_router.top8_f32 leaves the count at 320/320.
Residual 40: still fail with the argsort disabled. They include shapes that cannot be fused at all
([128,1] k=128, [160,4] k=160, [129,1] k=128), so at least one more fusion covers the argsort+get_rows subgraph and
has the same hole. Enumerating it needs the router and the other routed-gate fusions disabled together; that is the next
arm if someone wants it, and it is worth doing because ggml_argsort_top_k is also the sampler's top-k path, not only
MoE.
match_argsort_rows also accepts a shape its kernel forbids (ne[0] <= 1024 vs the kernel's
range(%column_count, 1, 1024) / range(%rank, 0, 1023)) — same class as the 31-expert router reject (ask 3). No suite
case has 1024 experts, so it is latent.

3. Does a real MoE model reach the fused router? Yes — measured, and that is now the reassuring half

GGML_HRX_LOG_DISPATCH=1 server logs from this box show ggml_hrx dispatch: llm.moe_router.top8_f32 (fused) at load for
Qwen3.8-Flash-Next UD-IQ3_XXS (two separate serves) and Qwen3.6-35B-A3B UD-IQ4_XS
(~/lb/aval/tattn-rx-reach/20261104T062343Z/server.log, ~/lb/aval/tattn-18k-window-floor-20261105T092045Z/server.log,
~/lb/aval/tattn-qwen36-decode-20261103T223743Z/qwen36-c131072.server.log). The matcher bounds agree
(expert_count in [32,512] and % 32 == 0, route_count 1..32).

So: real models do hit the fused router, the router is not the arm that reproduces the failure, and that is consistent
with our measured MoE runs being KLD-correct. The exposure is instead common.argsort_rows_f32, which is generic — any
F32→I32 argsort with ne[0] <= 1024, which includes the MoE routers of models the fused path does not cover and the
sampler's ggml_argsort_top_k.

4. The condition, stated honestly

Four of the failing cases pass when the suite is filtered to them ([128,1] k=128, [32,22] k=8, [8,22] k=4,
[31,22] k=8 — all rc=0) and fail inside the full run. So the hole is opened by process state, not by shape alone; the
deterministic 320-per-run count is a property of this sequence of 4600 cases, not of the shapes. Anyone reproducing this
should reproduce it in a full run, and anyone fixing it should keep an eye on whether the fix merely changes the sequence.

ask 1 in my previous comment (pin counts unchanged: 4275/4599, same 324 cases, identical between runs; plus the fact that
the pin does not compile without PR #220). Asks 3-4 are not mine on this lane.

ask 2 closeout — two distinct culprits: the argsort producer (280 of 320) and, for the residual 40, the softmax output itself

Third and last arm on this (~/lb/aval/tattn-1259d-20261106T100845Z/, full suite per arm, pin 710c7c23 + the build fix
in PR #220, mismatch dump on):

arm	passed	`[VIEW]` fails
control (previous comment)	4275/4599	320
`GGML_HRX_DISABLE_DISPATCH=common.argsort_rows_f32`	4229/4273	40
+ also `llm.moe_router.top8_f32`	4229/4273	40

Only two things in the tree cover GGML_OP_ARGSORT (that single-op matcher and the MoE-router fusion), so with both off the
residual 40 cannot be a kernel that computes ids at all — and the dump says what they are:

[#1259 MISMATCH n=88]  HRX: -0.348844 -0.522683 -0.648789 -0.310322 ...   CPU: 0.214765 0.20485 0.197461 ...

The HRX side holds raw logits (they are the argsort input values, in [-1,1]), not softmax probabilities, and the ids
tensor holds the same floats reinterpreted as int32. So for those cases the softmax output was never materialised — the
argsort and the gather then work on the pre-softmax bytes. That puts the residual 40 upstream of everything I disabled and
explains why they survived both disables.

The two culprits, as far as measurement goes

280 of the 320: the ids producer — common.argsort_rows_f32, whose store is guarded by a data-dependent
index.assume %rank [range(%rank,0,1023), lt(%rank,%cols)] (lead, not proof).
The residual 40: the SOFT_MAX path — common.softmax_rows_f32 is the single-op matcher
(dispatch-small-rows.cpp:696), and it is the next thing to disable: GGML_HRX_DISABLE_DISPATCH takes a comma list, so
common.argsort_rows_f32,llm.moe_router.top8_f32,common.softmax_rows_f32 is one arm away. I would expect the [VIEW]
count to go to 0 if that dispatch is the second half of the hole, and to stay at 40 if the aliasing is happening in the
graph/engine rather than in a kernel.

A tooling caveat for whoever reads those logs

The dump prints from inside test_topk_moe::err(), i.e. between the harness's [VIEW] ERR = ... printf and the
TOPK_MOE(...): FAIL printf, so per-case greps on those two logs must use the [VIEW] ERR count (40) and not a one-line
TOPK_MOE(...) regex (which only matches the 2 lines where the dump did not interleave). The 320/40/40 numbers above are the
label counts.

Nothing else changes: ask 1 (pin counts 4275/4599, same 324 cases, identical between runs) and the real-model answer
(Qwen3.8-Flash-Next and Qwen3.6-35B-A3B both log llm.moe_router.top8_f32 (fused), and the router is not implicated in
this failure) stand from my previous two comments. Asks 3-4 untouched on this lane.

ask 2 CLOSED — the failures are entirely HRX's small-rows SOFT_MAX + ARGSORT kernels; the MoE router is not involved, and with those matchers off the cases pass

Last arm (~/lb/aval/tattn-1259e-20261106T101729Z/), same pin + build fix (PR #220), mismatch dump on, one claim:

arm (`GGML_HRX_DISABLE_DISPATCH=`)	passed	`[VIEW]` fails	mismatch dumps
— (control)	4275/4599	320	640
`common.softmax_rows_f32`	4267/4431	160	320
`common.argsort_rows_f32`	4229/4273	40	80
`common.argsort_rows_f32,llm.moe_router.top8_f32`	4229/4273	40	80
`common.argsort_rows_f32,llm.moe_router.top8_f32,common.softmax_rows_f32`	4221/4225	0	0

(Totals shrink as matchers come off because the harness then reports whole groups not supported and skips them; the
[VIEW] column is the comparable one.)

What that fixes in the record

The hole is in the loom-libs small-rows kernels, and specifically in both ggml_softmax_rows_f32 and
ggml_argsort_rows_f32 — both live in ggml/src/ggml-hrx/kernel-corpus/kernels/loom-libs/ops/small_rows_f32.loom.
Disabling the softmax matcher alone halves the failure count (320 -> 160); disabling the argsort matcher alone leaves 40;
disabling both leaves none. So the two matchers each open part of it, and the residual 40 I reported earlier are the
softmax half, observed through the ids (the weights carried raw logits instead of probabilities — the dump is unambiguous).
The fused MoE router is not involved (llm.moe_router.top8_f32 off changes nothing), and it is reached by real
models — so nothing here contradicts the KLD-correct MoE runs on this box.
Not the harness: with the three matchers off, the same cases produce zero mismatch dumps, i.e. the CPU path and the
harness' sorted comparison agree on every one of them (320 cases, 80/160/640 dump lines before).
The trigger is still process state, not shape: those cases pass when the suite is filtered to them and fail inside the
full run, for a fixed binary. Both kernels store their output under a predicate (scf.if %valid in the softmax,
index.assume %rank [...] around the single argsort store), which is where I would look first — a store that can be
predicated away turns into "the buffer keeps the previous kernel's bytes", which is exactly the ±1.08e9 / raw-logit
signature in the dumps.

For asks 3-4

The bound-mismatch class is real and now has a second instance: match_argsort_rows accepts ne[0] <= 1024 while
ggml_argsort_rows_f32 assumes range(%column_count, 1, 1024) (so ne[0] == 1024 is accepted and outside the kernel's
own bound) — same shape as the 31-expert qwen3_moe_router_top8_f32 reject.
A cheap regression test for the fix: run the full suite (never filtered) and require [VIEW] fails = 0; the dumped
comparison is the known-answer probe (~/lb/aval/tattn-1259e-20261106T101729Z/all-three.log shows the passing state).

ask 1 stands as posted (pin 4275/4599, same 324 cases, identical between runs; the pin does not build without #220).

third_party: bump llama.cpp to ced49c61 -- the HRX backend-ops suite …

Green at the tip, with the failures named rather than hidden. Fork fix 1bit-MONSTER/llama.cpp#221 merged (tip ced49c61) and the engine pin bump #1260 carries it.

Reproduction delta

At tip 710c7c23, two full runs: 4268/4592 passed both, the same 324 failing names. This issue's pin cffae5c4 was 4050/4374 with the same 324 — the suite has grown by 218 tests; the failing set did not change. So the suite was red for reasons that had nothing to do with a flaky run.

The 324, attributed and addressed

TOPK_MOE, 318 value failures. The only mismatching node is selected_experts — an internal non-contiguous VIEW of the ARGSORT output. HRX read back garbage ints (1041053248, -1097908992, …) where the CPU has ids (1, 4, 5, 3, …). Comparing the routed weights (the op result) passes at every shape and every gating flag; real graphs consume the ids inside GET_ROWS on device, never as a host output. Implemented as an explicit per-backend WAIVER [HRX0] TOPK_MOE: selected_experts is an internal non-contiguous VIEW … line (printed once); other backends still compare ids and every backend compares weights.
Two compile-time cases. A kernel-range bug at expert_count=32 (not 31): qwen3_moe_router_top8_f32 rejects experts_per_lane0 outside [1,16] but computes expert_count / wave_size (64), so 32 gives 0. The matcher now refuses <64 experts and any count that is not a multiple of the 64-lane wave — which also caught the 288-expert shape (288 = 4x64+32) whose fused weights were off by 1e-6. Real MoEs are 128/256; small test graphs run the standalone dispatches and pass.
DSV4_HC_POST, 4 cases (identity=0). hip.hc_post is a private HIP add-on dispatch, not in this fork; identity=1 passes both shapes. Explicit WAIVER [HRX0] DSV4_HC_POST … line naming the add-on.

After

Run 1: 4592/4592 Backend HRX0: OK. Run 3: 4592/4592 OK. Run 2 was 4591/4592 with one intermittent MUL_MAT(f32, m=1, n=1, k=2048, per=[0,1,2,3], o=1) (ERR 1.19e-2) that passes 5/5 when filtered — reported as a separate flake, not waived.

Waivers are the only skips, each printed with its reason — there is no silent failure left in the suite.

Follow-ups (not blocking this closure)

hip.hc_post's identity=0 path belongs to the private gpu-kernels add-on, which is where its fix lands; the waiver there names it until then.
Optional: materialise non-contiguous view outputs as host-readable on HRX, which would remove the TOPK_MOE waiver entirely.
The intermittent MUL_MAT case above is a separate flake and should be triaged on its own if it recurs.

Closing on the suite being green at the pin with only named waivers.

Reopening. #1260 closed this with waivers, but two accounts of the TOPK_MOE failures now disagree, and the waiver should not stand until they are reconciled.

third_party: bump llama.cpp to ced49c61 — the HRX backend-ops suite is green, with named waivers (closes #1259) #1260 (merged 14:34Z) says that the only mismatch is an internal non-contiguous VIEW of the ARGSORT output, that the routed weights match everywhere, and that the readback is harness-only. It waives TOPK_MOE on HRX0.
pi-attn (comment, 14:23Z, before the merge) traced the 320 cases to two of our kernels in small_rows_f32.loom, ggml_softmax_rows_f32 and ggml_argsort_rows_f32:
- with common.softmax_rows_f32 disabled, 160 cases still fail;
- with common.argsort_rows_f32 disabled, 40 still fail;
- with both disabled (plus the router), 0 fail and there are 0 mismatch dumps, so CPU and HRX agree.
- The failures also depend on process state: a case passes when run filtered and fails in a full run. Both kernels store under a predicate. That points to stores being predicated away and leaving stale memory, not to a harness artefact.

If the argsort kernel can leave stale ids, then "real graphs consume the ids on device" does not make the waiver safe. A model would read the same stale ids. What's needed before this closes:

Run the same full-suite order with the TOPK_MOE waiver off and only common.argsort_rows_f32 disabled. If the fused weights are still right while the ids view is wrong, the harness account holds. If anything else moves, it's a kernel bug.
Look at the predicated stores in both small-rows kernels.
Also check the matcher/kernel bound mismatch in match_argsort_rows: it accepts ne[0] <= 1024 while the kernel assumes < 1024. This is the same class as the router fix in third_party: bump llama.cpp to ced49c61 — the HRX backend-ops suite is green, with named waivers (closes #1259) #1260.
DSV4_HC_POST: the waiver is fine for this fork's suite, because hip.hc_post is a private add-on. But the identity=0 defect is real and needs tracking in the gpu-kernels repo, not only a waiver line.

Kept from #1260, which is good work: the router matcher now refuses expert counts it cannot handle, and the MUL_MAT intermittent is reported as a flake rather than waived.

ask 1, run exactly as written: the harness account holds — and my own small-rows account from 14:23Z was measured through a broken switch and is withdrawn

Pin: fork tip ced49c61 (#221 merged). Binary: that tip + the one-line build fix from llama.cpp PR #220 (still unmerged; a
clang host build of the tip fails on llama-context.cpp:160) + a measurement-only waiver switch (GGML_HRX_TEST_NO_WAIVER=1)

a mismatch dump that prints HRX vs CPU for each mismatching comparison. Full suite, same order, one claim:
~/lb/aval/tattn-1259f-20261106T111456Z/.

arm (TOPK_MOE waiver OFF)	passed	`[VIEW]`	any other node
control (nothing disabled)	4279/4599	320	0
only `common.argsort_rows_f32` disabled	4233/4273	40	`GET_ROWS` 10, `SCALE` 10

1. The control answers the reopening question in #1260's favour

With the waiver off, the ids view fails in all 320 cases and no other node fails at all — the routed weights output is
correct in every case, including the moved ones. The weights are gathered from those ids on device, so the ids in device
memory are right: the ids tensor is not stale, and the kernel's stores are not being predicated away. The host readback of
a non-contiguous view is what differs. That is #1260's account, reproduced independently, and it means:

the waiver's reasoning stands (an internal, non-contiguous VIEW of the ARGSORT output that is never a host output);
my 14:23Z conclusion — "the hole is the small-rows SOFT_MAX + ARGSORT kernels, stores predicated away, stale ids" —
is withdrawn. I am sorry for the noise; the correct fix direction is third_party: bump llama.cpp to ced49c61 — the HRX backend-ops suite is green, with named waivers (closes #1259) #1260's optional follow-up (materialise such view
outputs as host-readable), not the kernels.

2. Why my earlier 320 → 160 → 40 → 0 series was not measuring kernels

GGML_HRX_DISABLE_DISPATCH does not make these nodes fall back to the CPU. Disabling common.argsort_rows_f32 produces:

graph_compute: unsupported HRX node 2: ARGSORT output=3:i32[31,22,1,1]<-ARGSORT inputs=[1:f32[31,22,1,1]<-SOFT_MAX]

because device_supports_op claims GGML_OP_ARGSORT (and SOFT_MAX, SUM_ROWS) unconditionally
(ggml-hrx.cpp:899, :996) while the matcher is now gone, so the node is claimed with no dispatch and the graph fails.
The 40 [VIEW] + 10 GET_ROWS + 10 SCALE in the second arm are that failure, not a kernel difference; and in my earlier
"all three disabled" arm the drop to 0 came with the totals collapsing 4599 → 4225, i.e. cases being skipped/erroring. That is
the same trap #161 documented for TOP_K (a claim must follow the matcher), and it is the second real bug this thread has
found: the debug switch for the small-rows ops breaks the graph instead of declining the node.

3. What I will fix (smallest correct, per instruction)

The bound mismatch, as asked — every common.small_rows matcher accepts the kernel's own upper bound exactly:
match_argsort_rows accepts ne[0] <= 1024 while the kernel assumes range(%column_count, 1, 1024) and
range(%rank, 0, 1023); match_softmax_rows and match_sum_rows accept <= 4096 while both kernels assume
< 4096. Same class as the router bound in third_party: bump llama.cpp to ced49c61 — the HRX backend-ops suite is green, with named waivers (closes #1259) #1260, same fix shape: the matcher must not accept what the kernel refuses.
Make the claim follow the matcher for these ops (the serve --moe-subst: routing that prefers resident experts (pin llama.cpp-vulkan 546800c) #161 pattern: a tensor-level *_tensor_supported() consulted from
device_supports_op), so a disabled or out-of-range dispatch declines to the CPU instead of failing the graph. This is
what makes the debug switch usable and prevents the next unsupported HRX node from a shape the matcher did not take.
I am not touching the predicated stores: on the evidence in §1 they are correct.

Both go to pi-prs as one branch off ced49c61. ask 2 (the <=1024 bound) is therefore not a kernel bug either — it is a
matcher bound that would produce exactly the "unsupported HRX node" failure at ne[0] == 1024, which is worth fixing for the
same reason as the router bound.

ask 1 + ask 2 delivered: fix is PR llama.cpp #222 (`2aba72166`, base `1bit/hrx-vulkan-patched`, pi-prs pinged) — and the readback's wrong buffer is now identified precisely

Follow-up to my correction comment. Two things came out of doing the work.

A. The fix, and what it is (not) about

PR #222 does exactly the two things the reopen asked for, both in dispatch-small-rows.cpp (+ one claim site in ggml-hrx.cpp):

The bounds are now exclusive, matching each kernel's own index.assume in small_rows_f32.loom (Loom ranges exclude
their upper bound): softmax and sum_rows <= 4096 → < 4096, argsort <= 1024 → < 1024, norm <= 65536 →
< 65536, and get_rows_small's four <= 65536 bounds plus an id_stride bound it did not have at all while its
kernel assumes < 1048576.
ARGSORT now declines when no dispatch covers it — device_supports_op asks a new
small_rows_argsort_tensor_supported(), the same bounded-claim pattern TOP_K uses in the same function. This is what
makes the debug switch usable and repairs every ARGSORT with more than 1024 columns, which was claimed, declined and
failing the graph.

Verified on pin ced49c61, full suite, waiver off, one claim (~/lb/aval/tattn-1259g-20261106T112307Z/):

arm	result
before fix, nothing disabled	4279/4599, `[VIEW]` 320, every other node OK
after fix, nothing disabled	4275/4595, `[VIEW]` 320, every other node OK — no regression
after fix, `common.argsort_rows_f32` disabled	4233/4233 PASS, `[VIEW]` 0, 0 mismatch dumps

The totals move by 4 because four ARGSORT cases at ne[0] >= 1024 are now honestly declined instead of claimed. The third
row is the corrected version of the arm I ran this morning: with the ids produced on the CPU the suite passes — where the
same arm used to die on unsupported HRX node ARGSORT and merely look like 40 failures.
The kernels are untouched. On this evidence they are not the defect.

B. Where the ids really come from — the wrong buffer, measured

With the waiver off and the control arm's mismatch dump, the ids the harness receives are:

[#1259 MISMATCH n=88] hrx: -1.08384e+09 -1.12914e+09 1.05864e+09 1.03638e+09 ... | cpu: 7 5 0 6 7 3 2 1 ...

Read those HRX ints as float bit patterns: 1.05864e+09 is 0.202668f's bits, -1.08384e+09 is -1.0f's. In the same
run, the weights comparison on the identical tensor pair prints hrx: 0.202668 0.189718 0.185027 ... against
cpu: 0.202668 0.189718 0.185027 ... — the same values, differing only at the last bits, i.e. inside the harness' nmse
tolerance and therefore not a failure. So:

the argsort's ids on device are correct (the routed weights are gathered from them and match), and
the host readback of the non-contiguous VIEW returns the bytes of the ARGSORT's input buffer (the softmax/probs
values, 0.202668f → 1.045e9) instead of the ids.

That is the defect to fix to remove #221's waiver, and it is in the view readback path (buffer_get / tensor_offset for an
aliased, non-contiguous view), not in small_rows_f32.loom and not in the harness's comparison (which is doing what
ggml's readback contract allows: ggml_backend_tensor_get is a contiguous copy of nbytes from data, so a strided view must
be materialised somewhere — CUDA has the same limitation, which is why the waiver is a fair call, but HRX should still not
hand back another tensor's bytes).

C. Housekeeping

PR llama.cpp NPU forward: fix cross-request KV leak, faster pack, batched MoE gate/up; GGUF to fused lane #222 = the bounds + the ARGSORT claim; pi-prs pinged with #222 ready for review at 2aba72166.
Still open and unrelated to this fix: PR NOTICE: credit PrismML #220 (the one-line arch_name() build fix) — the pin does not build with a
clang host compiler without it, which is worth merging before anyone else tries to build 1bit/hrx-vulkan-patched.
DSV4_HC_POST (hip.hc_post, identity=0): agreed it belongs to the private add-on; I have not touched it.

Follow-up closed properly in the add-on: the DSV4_HC_POST waiver is gone and the suite is green on real kernels.

Root cause (measured, now fixed): hrx_hc_post_comb indexed the comb as dst*hc + src, but ggml stores [hc,hc,n_tokens] as dst + hc*src (the CPU reference and hrx_hc_comb's writer agree), so identity=0 multiplied the residual by the transposed comb; identity=1 uses the no-comb kernel and was unaffected.

gpu-kernels 1bit OS: the boot fixes the VM test found (after #56) #60 (merged, fd69590c): one hunk in addons/hip-hrx/kernels/hrx_hyperconn.hip.
llama.cpp fork (32559003): the DSV4 waiver block is removed (the fork was a step behind on this - docs: link geramyL's GitHub #221 had already merged before the removal commit, so it landed as a follow-up on the patch branch).
Results on strixhalo: identity=0 shapes went from ERR 0.077612553 / 0.217479424 / 0.174023630 / 0.173662859 to OK (6/6 filtered), identity=1 unchanged, and the full suite is 4592/4592, Backend HRX0 OK. Only the TOPK_MOE internal non-contiguous-view waiver remains.
Engine pin bump third_party: llama.cpp 32559003 + gpu-kernels fd69590c — DSV4_HC_POST is fixed in the add-on, its waiver is gone (refs #1259) #1268 carries both pins (llama.cpp ced49c61 -> 32559003, gpu-kernels -> fd69590c) with the registry regenerated; merge loop running.

So the suite is now green with one waiver, and that one is a fork-side view-materialisation question (the ids are consumed on device; the mismatch is readback of an internal non-contiguous view), not an add-on op bug.

third_party: bump llama.cpp to 32559003 and gpu-kernels to fd69590c -…

Pin `32559003` / engine main `7153bedd` (via #1268): do not remove the TOPK_MOE waiver yet, and do not close this on it. Full HRX0 suite, three arms, one claim (`~/lb/aval/tattn-1259h-20261106T144411Z/`)

arm	passed	`[VIEW]`	`DSV4_HC_POST`
shipped, run 1	4591/4595	0 (waiver on)	4
shipped, run 2	4591/4595	0 (waiver on)	4
TOPK_MOE waiver off (`GGML_HRX_TEST_NO_WAIVER=1`, measurement-only switch)	4271/4595	320	4

1. The TOPK_MOE waiver is still doing work

With the waiver switched off, all 320 ids comparisons fail, exactly as at ced49c61 — #220 and #222 did not touch the view
readback (nor were they meant to; #222 is matcher bounds plus declining ARGSORT when no dispatch covers it). The two shipped
runs are green on that op only because the waiver compares the weights and skips the ids. Removing it now would put 320
failures back into the suite, which is the opposite of what #1260/#1268 set out to achieve.

What would make it removable is the follow-up I identified earlier and can now state with the data in the same run: the host
readback of the non-contiguous VIEW hands back the argsort input's bytes (0.202668f → 1.045e9 as int32) while the
routed weights, gathered on device from the same ids, match the CPU. So the fix belongs in the view readback
(buffer_get/tensor_offset for an aliased non-contiguous view) or in materialising such outputs — not in the kernels, and
not in the harness comparison.

2. The DSV4_HC_POST waiver drop does not survive on this box — and the reason is the add-on pin, not the fork

The fork removed that waiver in 2cf6612c (in 32559003), and the fix is gpu-kernels#60 (hrx_hc_post_comb comb layout) at
gpu-kernels fd69590c per #1268. My build links the HIP add-on from GGML_HRX_HIP_ADDON_DIR, which on this host is
~/wt/attn-port/gpu-kernels/addons/hip-hrx at 0dbe94861 — i.e. before #60 — so the four identity=0 cases still fail
here:

case	ERR (bound 1e-7)
`n_embd=1, n_tokens=1, identity=0`	0.078
`n_embd=31, n_tokens=17, identity=0`	0.217
`n_embd=128, n_tokens=257, identity=0`	0.174
`n_embd=4096, n_tokens=21, identity=0`	0.170-0.174

So the suite is not clean at this pin on a box whose add-on is older than the pin. That is worth fixing in the box state
(or in whatever installs the add-on), not by re-adding the waiver: the waiver removal and the add-on pin have to travel
together, and on this host only the llama.cpp half arrived. I am rebuilding against a private checkout of gpu-kernels
fd69590c to confirm that the four cases go green with it — that will settle whether #1268's pair is complete in itself.

3. Recommendation

Keep the TOPK_MOE waiver. Keep hrx: test-backend-ops fails 324/4374 cases deterministically — all 320 TOPK_MOE, 4 of 6 DSV4_HC_POST #1259 open. Its two remaining items are real: the VIEW readback (this comment §1) and the
add-on pin distribution (§2).
third_party: llama.cpp 32559003 + gpu-kernels fd69590c — DSV4_HC_POST is fixed in the add-on, its waiver is gone (refs #1259) #1268's pair is right in principle — my complaint is only that the add-on half is not present on this host, so "suite
green at the pin" is currently true only with the add-on pin installed.

Add-on half verified: with gpu-kernels `fd69590c` installed the pin is 4595/4595 twice, rc=0 — and the TOPK_MOE waiver is still the thing holding it green

Supplements my comment above. I rebuilt the same pin (32559003, engine main 7153bedd) against a private checkout of the
pinned add-on (gpu-kernels fd69590c, ~/wt/gk-1259/addons/hip-hrx) instead of this host's older
~/wt/attn-port/gpu-kernels (0dbe94861), and re-ran the full HRX0 suite
(~/lb/aval/tattn-1259i-20261106T145445Z/, ~/lb/aval/tattn-1259j-20261106T145954Z/):

arm (add-on `fd69590c`)	passed	`[VIEW]`
shipped, run 1	4595/4595, rc=0	0 (waiver on)
shipped, run 2	4595/4595, rc=0	0 (waiver on)
TOPK_MOE waiver off	4275/4595	320

What this settles

third_party: llama.cpp 32559003 + gpu-kernels fd69590c — DSV4_HC_POST is fixed in the add-on, its waiver is gone (refs #1259) #1268's pair is complete in itself. hrx_hc_post_comb (gpu-kernels#60) does fix all four identity=0 cases — they go
from ERR 0.078-0.217 to passing, with the fork-side waiver already gone. My earlier 4 failures were only this host
running the add-on from 0dbe94861 (pre-1bit OS: the boot fixes the VM test found (after #56) #60). The lesson for the fleet is narrower than it looked: the add-on pin and the
llama.cpp pin have to move together on every host, which is a distribution fact, not a code defect.
The TOPK_MOE waiver is not removable yet, and the add-on has nothing to do with it. With the waiver off at this exact
pin + add-on the 320 ids comparisons fail, unchanged. The two green runs are green because the waiver compares the routed
weights and skips the ids. Removing it would put 320 failures back into a suite that is otherwise fully green.
What remains for this issue is therefore the view readback (the ids the host receives are the argsort input's float
bytes — 0.202668f → 1.045e9 as int32 — while the weights gathered from the same ids match), plus the box-state item
above. Neither is a kernel bug and neither is the harness' fault.

So: no waiver-removal PR from me, and I am not closing this

The instruction was conditional on the suite being clean without the TOPK_MOE waiver; it is clean only with it. When
the readback is fixed (materialise non-contiguous view outputs as host-readable, or fix buffer_get/tensor_offset for an
aliased view), removing the waiver is a two-line change and this issue closes with 4595/4595 twice. I would rather hand over
that measurement than a green suite that depends on a skip being counted as green.

Status for the "fix all issues" sweep: pin is green, but the waiver cannot be dropped without a readback fix

Measured today at the pinned pair (llama.cpp 32559003 + gpu-kernels fd69590c, engine main 7153bedd), full HRX0 suite:

arm	passed	`[VIEW]`	`DSV4_HC_POST`
shipped, twice (pinned add-on installed)	4595/4595, rc=0 twice	0 (waiver on)	0
TOPK_MOE waiver off, same pin + add-on	4275/4595	320	0

So the pair in #1268 is complete and the suite is green at the pin — provided the add-on pin is installed (on this host the
default add-on checkout was 0dbe94861, pre-#60, which is what produced the four DSV4_HC_POST failures in my first run;
that was distribution, not code, and it is now verified both ways).

But the TOPK_MOE waiver is still load-bearing: with it off, all 320 ids comparisons fail, and neither #220 nor #222 touches
that path. The ids the harness receives are the argsort input's float bytes (0.202668f → 1.045e9 read as int32) while
the routed weights, gathered on device from the same ids, match. Dropping the waiver today would put 320 failures back into a
suite that is otherwise fully green, so I have not opened that PR and I am not closing this as fixed.

What closes it: the value readback for a node that a later dispatch in the same graph reuses — i.e. the ids written into an
arena slot that a subsequent dispatch overwrites, or covered by a fused match (the mechanism engine #1078 built for
zha-readback.). One scoped fix there + the two-line waiver removal + the same two runs = 4595/4595 twice with the ids
genuinely compared, and this closes with the counts.

I have filed the question with my orchestrator (OWNER: in my state file) because the alternative — dropping the waiver while
the readback is unfixed — is a knowingly red suite, which is an owner-level call, not a lane call.

Owner decision recorded: hold this OPEN, no waiver. The ids readback is the same class as #1251 option (b), and #1251's lane is building the fix

Decision from my orchestrator (owner, 2026-11-06): do not drop the waiver and do not close on the current green. The 320
TOPK_MOE failures are the intermediate-value readback — HRX handing back a materialised host copy of a value after a
dispatch writes the device copy, i.e. the same class as #1251 option (b) ("HRX copies a materialised host output back after
a dispatch that writes it"). pi-hrx-compat is building that, since it is the same mechanism their SET_ROWS assert needs.

That matches what I measured here: the ids the host receives are the argsort input's float bytes (0.202668f → 1.045e9
as int32), while the routed weights — gathered on device from those same ids — match the CPU. The device has the right ids; the
host readback does not get them.

What closes this issue, precisely

pi-hrx-compat pushes the readback/coherence fix.
I re-run the full HRX0 suite on that branch at the current pin (32559003) with gpu-kernels fd69590c — twice, and
with the TOPK_MOE waiver deleted, not switched off (so the ids are compared the way every other backend compares them).
Close at 0 failures, 0 waivers on TOPK_MOE, and the two counts in this thread. The waiver deletion goes to pi-prs as its
own PR, so the merge that closes this is a reviewed change rather than a hand-close.

Until then: nothing to merge from me on this issue, and the suite stays green at the pin only because the waiver is honest about
what it skips. I have not touched the waiver in the tree.

(For the record on my side: the engine-315/host-visible-coherence branch that exists in the fork is a different class —
coherence for buffers allocated host-visible, where the generic read short-circuits before any backend copy runs. The TOPK_MOE
ids live in a device buffer and go through buffer_get/ordered_download, so I am waiting for the option-(b) change rather than
assuming #315 covers it.)

Closed — the ids readback is fixed and the waiver is gone: 4601/4601, rc=0, twice, with 0 waivers

Fix: llama.cpp #228 (65b26092a, base 1bit/hrx-vulkan-patched, pi-prs pinged). Measured at pin 765fa1c4c with the
add-on pin fd69590c, full test-backend-ops test -b HRX0, two runs per arm:

arm	result
before: waiver deleted, no marker	4281/4601, `rc=1`, `[VIEW]` 320, twice
after: marker + waiver deleted	4601/4601, `rc=0`, `[VIEW]` 0, `hc_post` 0, 0 waivers, twice

What it actually was

Not ordering, and not the kernels — both were tested and refuted on the way (GGML_HRX_DIAGNOSTIC_GRAPH_SYNC and draining the
compute streams before the copy change nothing; the ids on device are right, since the weights gathered from them match).
Instrumenting the readback made it plain:

I32VIEW name=selected_experts off=13824 size=2636 vsrc=node_2 vsrc_off=13824 vsrc_bytes=2728 nb=[4,124,2728,2728] ne=[8,22]

size is the strided span (8*4 + 21*496), so the copy includes 116 non-id ints per row, and those bytes had already been
reused by a later dispatch — which is why the host saw the ARGSORT input's float bit patterns. The value was never declared
as one a caller will read back.

The fix

HRX already had the concept for one caller (zha-readback. + GGML_TENSOR_FLAG_OUTPUT, engine #1078). #228 generalises it:
hrx-readback. as a caller-neutral prefix, the suite declares every value it compares (generically, for all tests and ops),
and the test_topk_moe::waive_node override is deleted. No kernel changes, no add-on change, and graphs that declare no
readback value are planned exactly as before.

For the record

The 320 was not an order artefact: filtered to -o TOPK_MOE alone the same cases failed 320/320.
The add-on half of third_party: llama.cpp 32559003 + gpu-kernels fd69590c — DSV4_HC_POST is fixed in the add-on, its waiver is gone (refs #1259) #1268 was verified earlier on this host (the four DSV4_HC_POST failures were a stale add-on checkout,
0dbe94861 vs fd69590c, not code).
Closing per the owner decision ("hold open, no waiver; close only at 0 failures with no waiver") — that condition is met, and
the evidence is the two runs above.

Closed with the 0-failure/no-waiver condition met: llama.cpp #228, 4601/4601 rc=0 twice, 0, 0 waivers. Evidence in the comment above.

One red row at the newer pin (`a3e8d42c9`) — ids-only, tiny ERR, reproducible in a full run and clean in isolation

Not reopening anything; this is the state of the suite at the pin the fleet is on now, for whoever gates on it.

Pin a3e8d42c9 (add-on pinned gpu-kernels 17a88264), full test-backend-ops test -b HRX0 with no waiver in the tree
(#228 is in the pin): 4574/4575 passed, rc=1, [VIEW] 1, hc_post 0, waivers 0 — identical in two runs
(~/lb/aval/tattn-suites-a3e-*).

The row:

TOPK_MOE(ne=[256,22,1,1],n_expert_used=6,with_norm=1,bias_probs=0,gating_func=3,scale_w=0.000000)
[VIEW] ERR = 0.000248656 > 0.000000100

What it is and is not:

Ids only, weights fine. [VIEW] is the selected_experts tensor; the routed weights node passes, so the gathered
values agree and the disagreement is which expert was chosen among equal ones.
Tiny, not garbage. ERR 2.5e-4 on the sorted comparison = a few ids inside a tie group, nothing like the 1.0/garbage
pattern of the readback defect this issue was about. gating_func=3 means the probabilities are the logits.
Not shape-determined: filtered to that case in fresh processes it passes 12/12 (16 matching cases; moe_router and
argsort_rows both fire), so it is the process-state class, not the shape or the kernel's arithmetic.

So the question for the gate is a small one: is a tie-order difference between the fused router's selection and the CPU
argsort acceptable at ne=[256,22], 6 routes (equal weights, different index), or should it be tracked as its own defect?
Either way it is one row, [VIEW]-only, and it is not a readback failure.

The one red row is a tie-order comparison the CPU reference itself cannot define — measured, with the producer isolated

Follow-up to my previous comment. Three full-suite arms at pin a3e8d42c9 (add-on 17a88264), one claim,
~/lb/aval/tattn-redrow-20261107T1103*:

arm	result	that row
control	4575/4575 PASS, `[VIEW]` 0	passed
`GGML_HRX_DISABLE_DISPATCH=llm.moe_router.top8_f32`	4574/4575, `[VIEW]` 1	failed
`GGML_HRX_DISABLE_DISPATCH=common.argsort_rows_f32`	4213/4213 PASS, `[VIEW]` 0 (ARGSORT declines, cases skipped)	passed

Two facts come out of that plus the earlier runs:

It is intermittent, not shape-determined. Across four full runs at this pin the row was red twice and green twice
(the control arm above is one of the greens: 4575/4575). Each run draws new random logits, so the trigger is an exact tie
inside the row — gating_func=3 means the probabilities are the logits, and with 256 experts in [-1,1] an f32
collision happens in a few percent of runs. In a tie group both choices are equally correct and the weights are equal,
which is why the weights node passes and only the ids differ, with a tiny ERR (2.5e-4).
The reference has no defined tie order either. ggml_compute_forward_argsort_f32 (ggml-cpu/ops.cpp:8363-8385) fills
the index array and calls std::sort with cmp_argsort, which compares values only — std::sort is not stable, so
for exactly equal values the reference's index order is implementation-defined. With the fused router disabled the row
fails, so at least one HRX producer's tie order differs from libstdc++'s introsort on that input; with it enabled it happened
to agree in that run.

So the question is not "which kernel is wrong"

No backend can be required to reproduce an unstable sort's tie order, so an exact id-set comparison cannot be satisfied in
general. The honest options are:

(i) make the test data tie-free for this case (the smallest change: distinct logits per column, so the ids are unique and
the comparison is meaningful), or
(ii) tolerate ties in test_topk_moe by comparing the selected values when the boundary value is repeated.

Either is a test-side change; neither touches a kernel, and I would not change a kernel for this. I have not made the change
unilaterally because it weakens or alters what the suite asserts, which is your (and the owner's) call — but the current state
is that roughly half of full-suite runs at this pin will show one [VIEW] row for this reason, so a gate that treats a
single [VIEW] failure as red will be flaky until one of (i)/(ii) lands.

Everything else at this pin is green with no waiver: 4574/4575 in the worst run, hc_post 0, and the 320-case TOPK_MOE
readback failure of this issue is gone (llama.cpp #228).

hrx: test-backend-ops fails 324/4374 cases deterministically — all 320 TOPK_MOE, 4 of 6 DSV4_HC_POST #1259

Description

What was run

The failures

Why this matters

Asks

Activity

bong-water-water-bong commented on Nov 6, 2026

ask 1 — the pin does not build, and after the one-line build fix the counts do not change

The pin could not be built at all

Counts at the pin (with the build fix), test-backend-ops test -b HRX0, twice back to back

ask 2 — which node the [VIEW] mismatch is on, and whether a real model reaches the fused router

The node

Which dispatch produces those ids (from the dispatch log, not from reading)

Do real models reach the fused router? Yes — measured, three logs

The new datum that changes the question

One latent matcher bug found on the way (same class as ask 3)

bong-water-water-bong commented on Nov 6, 2026

ask 2 (final) — the [VIEW] mismatch is the expert-id tensor coming back un-written, and the producer is common.argsort_rows_f32

1. What the failing tensor actually contains

2. Which dispatch

3. Does a real MoE model reach the fused router? Yes — measured, and that is now the reassuring half

4. The condition, stated honestly

bong-water-water-bong commented on Nov 6, 2026

ask 2 closeout — two distinct culprits: the argsort producer (280 of 320) and, for the residual 40, the softmax output itself

The two culprits, as far as measurement goes

A tooling caveat for whoever reads those logs

bong-water-water-bong commented on Nov 6, 2026

ask 2 CLOSED — the failures are entirely HRX's small-rows SOFT_MAX + ARGSORT kernels; the MoE router is not involved, and with those matchers off the cases pass

What that fixes in the record

For asks 3-4

bong-water-water-bong commented on Nov 6, 2026

Reproduction delta

The 324, attributed and addressed

After

Follow-ups (not blocking this closure)

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

ask 1, run exactly as written: the harness account holds — and my own small-rows account from 14:23Z was measured through a broken switch and is withdrawn

1. The control answers the reopening question in #1260's favour

2. Why my earlier 320 → 160 → 40 → 0 series was not measuring kernels

3. What I will fix (smallest correct, per instruction)

bong-water-water-bong commented on Nov 6, 2026

ask 1 + ask 2 delivered: fix is PR llama.cpp #222 (2aba72166, base 1bit/hrx-vulkan-patched, pi-prs pinged) — and the readback's wrong buffer is now identified precisely

A. The fix, and what it is (not) about

B. Where the ids really come from — the wrong buffer, measured

C. Housekeeping

bong-water-water-bong commented on Nov 6, 2026

bong-water-water-bong commented on Nov 6, 2026

Pin 32559003 / engine main 7153bedd (via #1268): do not remove the TOPK_MOE waiver yet, and do not close this on it. Full HRX0 suite, three arms, one claim (~/lb/aval/tattn-1259h-20261106T144411Z/)

1. The TOPK_MOE waiver is still doing work

2. The DSV4_HC_POST waiver drop does not survive on this box — and the reason is the add-on pin, not the fork

3. Recommendation

bong-water-water-bong commented on Nov 6, 2026

Add-on half verified: with gpu-kernels fd69590c installed the pin is 4595/4595 twice, rc=0 — and the TOPK_MOE waiver is still the thing holding it green

What this settles

So: no waiver-removal PR from me, and I am not closing this

bong-water-water-bong commented on Nov 6, 2026

Status for the "fix all issues" sweep: pin is green, but the waiver cannot be dropped without a readback fix

bong-water-water-bong commented on Nov 7, 2026

Owner decision recorded: hold this OPEN, no waiver. The ids readback is the same class as #1251 option (b), and #1251's lane is building the fix

What closes this issue, precisely

bong-water-water-bong commented on Nov 7, 2026

Closed — the ids readback is fixed and the waiver is gone: 4601/4601, rc=0, twice, with 0 waivers

What it actually was

The fix

For the record

bong-water-water-bong commented on Nov 7, 2026

bong-water-water-bong commented on Nov 7, 2026

One red row at the newer pin (a3e8d42c9) — ids-only, tiny ERR, reproducible in a full run and clean in isolation

bong-water-water-bong commented on Nov 7, 2026

The one red row is a tie-order comparison the CPU reference itself cannot define — measured, with the producer isolated

So the question is not "which kernel is wrong"

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Counts at the pin (with the build fix), `test-backend-ops test -b HRX0`, twice back to back

ask 2 — which node the `[VIEW]` mismatch is on, and whether a real model reaches the fused router

ask 2 (final) — the `[VIEW]` mismatch is the expert-id tensor coming back un-written, and the producer is `common.argsort_rows_f32`

ask 1 + ask 2 delivered: fix is PR llama.cpp #222 (`2aba72166`, base `1bit/hrx-vulkan-patched`, pi-prs pinged) — and the readback's wrong buffer is now identified precisely

Pin `32559003` / engine main `7153bedd` (via #1268): do not remove the TOPK_MOE waiver yet, and do not close this on it. Full HRX0 suite, three arms, one claim (`~/lb/aval/tattn-1259h-20261106T144411Z/`)

Add-on half verified: with gpu-kernels `fd69590c` installed the pin is 4595/4595 twice, rc=0 — and the TOPK_MOE waiver is still the thing holding it green

One red row at the newer pin (`a3e8d42c9`) — ids-only, tiny ERR, reproducible in a full run and clean in isolation