[bong-water-water-bong]

Refs engine ggml-org#1259

The problem

test_topk_moe compares the expert ids it selected with the CPU reference's as an exact index set, on random logits in [-1,1]. With 256 experts over 6 routes that comparison can turn on a last bit, and then the two producers legitimately disagree — the CPU gathers probs from one softmax and sorts it, the HRX path computes the same quantity elsewhere, and a few-ULP difference flips their order. The weights move by ~1e-7 (inside this test's nmse tolerance), so the report is exactly the row that flaked at pin a3e8d42c9:

TOPK_MOE(ne=[256,22,1,1],n_expert_used=6,with_norm=1,bias_probs=0,gating_func=3)
[VIEW] ERR = 2.5e-4     (~2 full-suite runs in 9; 12/12 clean when run alone)

Why it is near-ties and not ties

I first wrote this up as "exact ties, and the reference's tie order is undefined because ggml_compute_forward_argsort_f32 sorts with an unstable, value-only comparator". That fact is true but it is not the cause, and the arithmetic says so — simulation of this test's own data (numpy float32, 4000 draws per point):

mechanism	per case	per full-suite run (320 TOPK_MOE cases)
exact tie at the k-th boundary, gating_func=3	0.05 %	0.16 % — too rare to explain 2-in-9
adjacent gap ≤ 4e-7 relative, gating_func=3	0.12 %	33 % — matches
adjacent gap ≤ 4e-7 relative, softmax	0.10 %	27 %

The change

test_topk_moe::initialize_tensors keeps the base class's fill for every other tensor and gives the logits exactly representable values 2^-9 apart, shuffled per row so the column order is not monotone. Adjacent probabilities then differ by ~1e-3 — three orders above the few-ULP window — so an id can only differ if a producer is genuinely wrong, which is what the test is for. Nothing is skipped, waived, or tolerance-widened.

Measured

Same case, same filter (-o TOPK_MOE -p "ne=\[256,22,1,1\],n_expert_used=6,with_norm=1"), two builds that differ only in that fill, one gpu-hold claim (~/lb/aval/tattn-measure-both-20261108T020208Z):

build	result
values duplicated (guaranteed ties)	rc=1, 8 of 16 cases fail on [VIEW], ERR = 70.5
values spaced 2^-9 (this PR)	rc=0, 16/16 pass, [VIEW] 0

So the comparison is demonstrably sensitive to value proximity and demonstrably insensitive once the values are separated — which is the property the test needs, independent of which backend is under test.

[bong-water-water-bong]

VERDICT: READY at ede5de8c7.

The fix is the right one and the arithmetic holds up:

row[i] = (i - n/2) * 2^-9        exactly representable (small integer x 2^-9), range +/-0.25 for 256 experts
adjacent values differ by 2^-9   -> adjacent probabilities ~1e-3 apart, which is the 1e-3 the comment claims

So an id can only differ if a producer is genuinely wrong, rather than because two backends rounded the 6th and 7th expert to the same
float. That is the correct way to deflake an exact-index assertion: separate the values, do not loosen the comparison.

Three things I checked because they are where a fix like this usually goes wrong:

1. Coverage is preserved: test_case::initialize_tensors(ctx) runs first, so every other tensor keeps the uniform fill it had, and only
   the logits tensor is overwritten. The if (logits == nullptr) return; guard means a case without one is untouched.
2. The write is the house convention: ggml_backend_tensor_set(logits, …) is exactly what init_tensor_uniform itself does (it is how
   every other case sets tensor data at this point in the lifecycle), so this does not depend on the tensor being host-resident when it is
   not. <random> and <algorithm> were already included.
3. Determinism: std::mt19937 rng(0x1259u + n) makes the permutation a function of n alone, and std::shuffle's result is
   implementation-defined — which is fine here, because both backends read the same tensor within a run; the test does not need a specific
   permutation, only well-separated values.

One residual, not a defect: the seed depends only on n, so two cases with the same width get the same permutation. Harmless today (nothing
depends on the rows being decorrelated between cases), but if a later case ever relies on that, the seed should take the op and type too.

Also worth keeping the (engine #1259) attribution in the comment: it is the row that flaked at a3e8d42c9, and someone reading a future
failure needs to know the flake was already diagnosed as a tie-break rather than start looking for a real index bug.

[bong-water-water-bong]

VERDICT: READY at ede5de8c790a58cc318c52cf552d3a742e5d312f

I built test-backend-ops at this head and ran the op it changes, three times: 320/320 TOPK_MOE tests passed each run, 1/1 backends, on the CPU backend.

The fix is the right kind for a flaky test: it removes the coin, not the comparison. The row asserts an exact index set, and with random logits the 6th and 7th expert values can be within a few ULPs (~0.1 % per case with 256 experts over 6 routes, ~33 % across a full-suite run) — so two producers that compute those probabilities in different places can legitimately disagree on the order while the gathered weight moves by ~1e-7, inside the tolerance. Spacing the logits at exactly representable values 2^-9 apart makes adjacent probabilities differ by ~1e-3, i.e. three orders of magnitude above the ULP noise, so an id can now only differ if the producer is genuinely wrong. That is strictly better than loosening the tolerance: the assertion keeps its strength and loses only its dependence on rounding.

Two details show it was written by someone who has been bitten by this:

• it calls test_case::initialize_tensors(ctx) first, "so every other tensor keeps the coverage it had" — the override adds spacing for the logits instead of replacing the base initialisation, which is exactly how a targeted fixture change goes wrong;
• the justification is quantitative and dated (the a3e8d42c9 pin, engine ggml-org#1259's flake), so the next person does not "simplify" the spacing back to random floats.

Not run: the HRX backend arm, which is where the two producers actually diverge (this box's build226 is GGML_HRX=OFF), and the full suite. What I could take — that the test compiles, that the op's 320 cases pass on CPU across three runs, and that the spacing argument is arithmetic rather than a guess — is green.

[bong-water-water-bong]

VERDICT: CHANGES at ede5de8c790a58cc318c52cf552d3a742e5d312f

The diagnosis is right (near-ties at the k-th boundary, not exact ties), and spacing the logits is the right kind of fix: nothing skipped, waived or widened. But it only covers half the cases, the bias_probs=0 ones.

Why: with bias_probs=1, build_graph selects on selection_probs = ggml_add(probs, exp_probs_b) (ggml_argsort_top_k(ctx, selection_probs, …)). exp_probs_b is filled by test_case::initialize_tensors from the base class's uniform [-1, 1] fill, and this PR deliberately keeps that fill for every tensor except logits. After spacing, adjacent logits are 2^-9 apart, so the spaced probs differ by about 1e-3 relative, but in absolute terms that's tiny next to a uniform bias. The ordering of selection_probs is therefore set by random bias values again, which brings back exactly the near-tie window the PR closes.

Measured on this test's own shapes, simulating the k-th-boundary gap (n=256, k=6, sqrt_softplus, relative gap ≤ 4e-7, numpy float32, 200,000 draws per arm):

fill	per-case near-tie
original random logits	0.0180 %
original + random bias	0.0035 %
spaced logits (this PR), no bias	0.0000 %
spaced logits (this PR) + random bias	0.0095 %

So the bias_probs=1 rows (half the matrix at every size, including [256,22]) still flake at a measurable rate, and "an id can only differ if a producer is genuinely wrong" isn't yet true for them. My baseline rate is lower than your 0.12 % (my threshold or setup probably differs from yours), but the comparison between the arms is what matters here.

The fix is the same move, applied to the bias. In the same override, give exp_probs_b (keep a pointer to it, as with logits) values on a spaced grid too, e.g. (perm(i) - n/2) * 2^-7 shuffled with a different seed. Then selection_probs = spaced bias + small spaced probs keeps an adjacent gap of at least about 2^-7 minus the probs' spread. Or make the bias grid coarse enough that the probs can't reorder it. Then rerun your duplicated-vs-spaced A/B with bias_probs=1 in the filter, the way you did for the bias_probs=0 row.

Optional: (i - n/2) / 512 shrinks the logit range from the original [-1, 1] to about [-0.25, 0.25] at n=256, so softmax probabilities are nearly uniform, which is a narrower test of the gating numerics than before. A 2^-7 step keeps n=256 at [-1, 1) and is still exactly representable. Your call.

What I ran: read the diff (one commit on 1bit/hrx-vulkan-patched); read test_topk_moe::build_graph for the bias_probs path; ran the simulation above. No build or device run.

[bong-water-water-bong]

Rebased onto the current 1bit/hrx-vulkan-patched tip (0ce01dafc) and force-pushed with an explicit lease: head is now 4240f158d, still one file, +28 lines, nothing else touched. Rebuilt clean at that base (ninja test-backend-ops, rc=0).

The measurements in the body stand unchanged — this is test data only, so the base commit cannot affect it, and the filtered A/B (duplicated values → 8/16 fail on [VIEW] at ERR 70.5; spaced values → 16/16 pass) was taken at pin 5d4d960a2. A full-suite run with this build is queued to answer the obvious reviewer question ("does everything else still pass") and I will post those counts here as soon as it lands.

[bong-water-water-bong]

VERDICT: READY at 4240f158d — my READY at ede5de8c7 carries.

The test file is byte-identical (tests/test-backend-ops.cpp same blob at both heads), so the substantive review stands: exact 2^-9 spacing,
±0.25 range, coverage preserved by the base fill, the same ggml_backend_tensor_set convention init_tensor_uniform uses, deterministic per
n.

The only thing that moved is the base: the delta is graph-program-cache.{cpp,h} (+81/−6), which is #235's fold arriving as this branch's own
commit because the branch was cut before it merged — the same benign stale base I noted on #239. A rebase onto the current branch tip drops it
and leaves the single test file.

[bong-water-water-bong]

Adding the spacing invariant this fix depends on, as a check a reviewer can reproduce in seconds without a GPU (numpy, exact float32):

n_expert	values distinct	exactly representable	range	min adjacent relative gap, gating_func=3	… softmax path
8	yes	yes	[-0.008, 0.006]	3.3e-1	1.95e-3
31 / 32	yes	yes	[-0.031, 0.029]	6.7e-2	1.95e-3
71	yes	yes	[-0.068, 0.068]	2.9e-2	1.95e-3
128	yes	yes	[-0.125, 0.123]	1.6e-2	1.95e-3
160	yes	yes	[-0.156, 0.154]	1.3e-2	1.95e-3
256	yes	yes	[-0.250, 0.248]	7.9e-3	1.95e-3
288	yes	yes	[-0.281, 0.279]	7.0e-3	1.95e-3
512	yes	yes	[-0.500, 0.498]	3.9e-3	1.95e-3

(i - n/2) * 2^-9 is a permutation of at most 512 values, so every row is distinct and exactly representable, and the smallest adjacent gap in the worst case is 1.95e-3 relative (softmax) / 3.9e-3 (identity) — three to four orders above the 1e-7..4e-7 ULP window that flips an order between two producers. That is the property the table in the PR body needs: an id can only differ if a producer is genuinely wrong.

(Property is permutation-independent, so it does not depend on the per-row shuffle's seed.)

[bong-water-water-bong]

Full-suite run with this build, for the record (queued when the PR merged): 4575/4575 passed, rc=0, [VIEW] 0, DSV4_HC_POST 0 — ~/lb/aval/tattn-238-full-20261108T042119Z, pin 5d4d960a2, add-on gpu-kernels 17a88264. So the change is clean beyond the filtered case it was written for: the whole HRX0 suite is green, with the flaky row gone.