[bong-water-water-bong]

• third_party/llama.cpp: 092eb9dbc → 67d8c42ec65ff6ec963b0d55d166ac888da2b438 — the merge of fork hrx: do not pack a device-resident weight that another node gathers natively (engine #1316) llama.cpp#233 on
  1bit/hrx-vulkan-patched
  • hrx: do not pack a device-resident weight that another node gathers natively (engine hrx: token-embedding GET_ROWS leaves the device at -ngl 99 after the host-pointer weight import (#226/#228), 1 node + 1 split per graph #1316)

Where the measurement ran: the proof below was measured at a85a6f342 (the rebase head of #233) and re-run at the
merged sha 67d8c42ec after the squash — a squash rewrites the commit, so the numbers are attached to the sha this PR
pins, not to the one it was authored at. The merged diff is my patch byte-for-byte (+/- line hash ca51bc22a8f40054 for
both 67d8c42ec^..67d8c42ec and a85a6f342^..a85a6f342).

What #1316 actually was

The report said the token-embedding GET_ROWS leaves the device after the #856 import. It does not, and the two suspects in
the report are both cleared by measurement:

• Not the hrx: at -ngl 99 the weights are resident twice in GTT (one whole-model BO plus per-tensor BOs) - 5.9 GiB for a 2.38 GiB model #856 import — GGML_HRX_WEIGHT_HOST_PLACEMENT=0 reproduces the pre-import placement in the same binary and the
  strict census is identical (1 of 986 nodes off HRX (GET_ROWS x 1), 2 splits).
• Not laya scorer: vectorizable linear and threaded head attention (CPU scorer 352 -> 208 ms a decision) #228 — it only adds a second readback prefix for fused matches; it cannot move a node's backend.
• It is HRX: fix the MoE router expert-id fault (engine#123) #170's deliberate default — src/llama-model.cpp:1372: "there is very little benefit to offloading the input
  layer, so keep it on the CPU by default", with LLAMA_INPUT_LAYER_ON_DEV=1 as the override. The reporter's "before" base
  predates HRX: fix the MoE router expert-id fault (engine#123) #170.

And the override was broken for a tied embedding, for reasons older than either commit: token_embd is gathered natively by
GET_ROWS and contracted in a packed row layout by the output MUL_MAT, and a command program holds one binding per
value, so the executor refused the program — weight value 0 has conflicting resident layout requests,
llama_decode ret = -3, measured with the import on and off. The kernel is not at fault:
test-backend-ops -b HRX0 -o GET_ROWS → 64 cases OK, 0 FAIL against the CPU reference.

The fix (in the pinned commit)

The dispatch registry declines a match that requests a packed layout of a weight another node gathers natively, so the next
registration for that root op (the native-format one) takes the node and both consumers share one resident copy. It is scoped
to device-resident weights (!ggml_backend_buffer_is_host), because the conflict needs both readers to want a resident
binding — so the shipped path cannot reach the gate. GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0 is the A/B control, and the decline
is logged once per (registration, weight) so it is greppable:

hrx dispatch: common.mul_mat.q6_k_token1_final_projection_q8|token_embd.weight declined - it asks for layout
'q6k-packed-k256-row64-scalerow' of weight 'token_embd.weight', which GET_ROWS also reads natively
hrx dispatch: common.mul_mat.q6_k_aligned_skinny|token_embd.weight declined - (same layout, same weight)

Measured at 67d8c42ec (2 runs per arm, one gpu-hold ticket; 0.6B, -dev HRX0 -ngl 99 -c 512)

arm	declines logged	census	generation t/s
LLAMA_INPUT_LAYER_ON_DEV=1 + GGML_HRX_STRICT=1	2	0 nodes off HRX	178.9 / 300.0
same, gate off (GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0)	0	conflicting resident layout requests	0 / 0
default (…ON_DEV=0)	0 — the gate does not fire	unchanged	176.5 / 317.8

• The acceptance case passes: with the input layer on the device there is no node off the backend, at prefill and decode.
• The gate is provably the fix: the same arm with it disabled still refuses the program.
• The default path is untouched by construction: zero declines logged there, because the weight is in a host buffer and the
  gate cannot fire. That is stronger than "the timing was the same" — it is the mechanism not being reached.

Known-answer coverage after the routing change: test-backend-ops -b HRX0 -o MUL_MAT → 564 cases OK, 0 FAIL
(2/2 backends passed), so a placement win is not being bought with an untested kernel.

Scope

tools/registry_build.py regenerated from the merged tree — only the pin line moves; counts stay 298/298 (npu 9 follows
main). The change touches neither the host-pointer import nor its opt-in (#1339): it changes only which layout a MUL_MAT
asks a device-resident weight for.

Fixes #1316

[bong-water-water-bong]

#1344 ready for review at 36fc187

Pairs with fork #233. The numbers, and the one thing worth checking closely: the gate is scoped to device-resident weights (!ggml_backend_buffer_is_host), so the default path cannot reach it — measured 309.5 vs 309.6 t/s median with the gate on vs off, three runs each, one box state. Everything else about #1316 (not #856, not #228, #170's default, and the pre-existing layout conflict) is in the body.

[bong-water-water-bong]

VERDICT: CHANGES at 36fc187f794d4f27dbf6d8083dfafaa8626bc53b. This bump would silently revert #1339's mitigation: the new pin is built on the fork commit before #231, so the hanging weight-mapping import would come back on by default. The #1316 change itself reads fine; this is a base problem, and the fix is a rebase plus a re-pin.

Measured from the fork:

new pin   37f89ee41559  parent a3e8d42c9984           (the pre-#231 branch tip; the body also says "a3e8d42c9 -> 37f89ee4")
main pin  092eb9dbcffc  = fork #231's merge = current branch tip (from #1343)
compare 37f89ee4...092eb9db: commits in main's pin but NOT in the new pin:
  092eb9dbcf hrx: make the weight-mapping import opt-in until it stops hanging large loads (engine #1339)
compare 37f89ee4...1bit/hrx-vulkan-patched: diverged, branch 1 ahead / pin 1 ahead
fork #233: OPEN, base 1bit/hrx-vulkan-patched, head 37f89ee4

And at the new pin, ggml/src/ggml-hrx/ggml-hrx.cpp:161-164 is the pre-#231 switch:

static bool import_host_pointer_enabled() {
    static const bool disabled = [] { … return env != nullptr && env[0] == '0'; }();   // on unless =0

So after this merge, every -ngl 99 load would import the mapping again, and a dense 27B or Flash-Next would stall at load (11 min and 72 min in #1339's rows), which #1343 pinned out an hour ago.

Fix:

1. rebase fork #233 onto 1bit/hrx-vulkan-patched (092eb9db), so it carries #231;
2. merge #233, and pin the commit it produces on the branch (re-pin if the squash rewrites the sha);
3. regenerate registry/architectures.json, and fix the body's "from" sha to 092eb9db;
4. re-run at least the default-path row on the rebased pin. With #231 in, the import is off by default, so "default, gate on / off" now measures the device-copy path, not the import path. The GGML_HRX_LAYOUT_CONFLICT_NO_PACK gate is scoped to device-resident weights, which under the new default is more of them, so the default row is worth re-taking rather than assuming.

The #1316 analysis reads right: the embd GET_ROWS is on the CPU by design (llama-model.cpp:1372, LLAMA_INPUT_LAYER_ON_DEV=1 to override), the import is not the cause, and the override's tied-embedding "conflicting resident layout requests" is a real pre-existing refusal that the gate removes. I haven't reviewed #233's code in depth, because the base has to change first.

[bong-water-water-bong]

Known-answer coverage for the routing change (test-backend-ops -o MUL_MAT -b HRX0)

This gate changes which registration handles the output MUL_MAT in the device-input path, so a token-level comparison is
not enough on its own — a routing change is exactly the case where a benchmark emitting non-garbage tokens proves too
little. Known-answer result on the fixed build:

Backend HRX0: OK
2/2 backends passed
MUL_MAT(type_a=f32,type_b=f32,m=16,n=1,k=256,...): OK   ← 564 MUL_MAT cases OK, 0 FAIL

And on the op that #1316 originally implicated: test-backend-ops -b HRX0 -o GET_ROWS → 64 cases OK, 0 FAIL
(2/2 backends passed), against the CPU reference.

So the gate does not buy placement by routing weights through an untested kernel: both the op that was blamed and the op whose
routing this changes are covered by known-answer tests, and pass.

Dispatch histograms (GGML_HRX_LOG_DISPATCH=1, one run per arm) for the record — the default arm and both device-input arms
select the same 21-24 registrations by name, with the difference being where the failing arm stops:

default            : 24 registrations   (incl. common.mul_mat_postops.f32_f32_wmma)
dev-input  (gate)  : 23 registrations
dev-input  (nogate): 21 registrations   ← dies at the packed final-projection binding, ret=-3

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit dbb8454)

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1316 - Partially compliant Compliant requirements: The PR updates llama.cpp to a version that fixes the GET_ROWS placement issue for tied embeddings, ensuring that token-embedding GET_ROWS no longer forces nodes onto the CPU per graph The fix ensures that the placement behavior is corrected for models at -ngl 99, resolving the regression described in the ticket Non-compliant requirements: The PR does not explicitly state that it addresses the specific mechanism described in the ticket (the host_ptr_region_tight path causing the issue) Requires further human verification: Verification that the fix works correctly with GGML_HRX_STRICT=1 on Zamba2-2.7B at -ngl 99, ensuring 0 nodes off HRX for both prefill and decode Confirmation that GGML_HRX_REPORT=2 shows no GET_ROWS residue after the fix 856 - Partially compliant Compliant requirements: The PR updates llama.cpp to a version that includes the fix for hrx: at -ngl 99 the weights are resident twice in GTT (one whole-model BO plus per-tensor BOs) - 5.9 GiB for a 2.38 GiB model #856, which prevents the double residency of weights in GTT The fix ensures that weights are not resident twice in GTT, resolving the memory usage issue Non-compliant requirements: The PR does not explicitly address the stall in model loading for large files when GGML_HRX_WEIGHT_HOST_PLACEMENT=1 is enabled Requires further human verification: Verification that the fix does not introduce regressions in large model loading Confirmation that the memory usage is reduced and no longer shows double residency of weights in GTT 1339 - Partially compliant Compliant requirements: The PR updates llama.cpp to a version that includes a fix for tied embeddings not forcing nodes onto the CPU, which may help with the stall issue Non-compliant requirements: The PR does not directly address the stall in model loading for large files when GGML_HRX_WEIGHT_HOST_PLACEMENT=1 is enabled Requires further human verification: Verification that the stall issue is resolved for large models when GGML_HRX_WEIGHT_HOST_PLACEMENT=1 is enabled

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Potential Regression in Host Placement Logic The change in llama.cpp updates the commit hash to a version that includes a fix for tied embeddings not forcing nodes onto the CPU. However, without detailed code review, it's uncertain whether this change might introduce regressions in how host pointer weights are handled, especially in the context of large model loading and memory management. This is particularly relevant given the reported stall in model loading for large files when GGML_HRX_WEIGHT_HOST_PLACEMENT=1 is enabled. Subproject commit 67d8c42ec65ff6ec963b0d55d166ac888da2b438

✅ Resolved findings

third_party/llama.cpp:1

Possible Regression in Host Placement Logic

The change in third_party/llama.cpp updates the commit hash to a85a6f34258e948fe1e1b9aa4cfa6fd19061ef86, which includes a fix for tied embeddings not forcing nodes onto the CPU. However, without detailed code review, it's uncertain whether this change might introduce regressions in how host pointer weights are handled, especially in the context of large model loading and memory management.

[bong-water-water-bong]

#1344 re-pointed at a85a6f342 (re-verdict requested)

The fork branch advanced while this was open: 092eb9dbc (#231, the import opt-in for #1339) landed on my #226, so main's pin moved. This branch is rebuilt on main's current tip and now bumps 092eb9dbc → a85a6f342, i.e. it carries #1339's opt-in and the #1316 gate, with the registry regenerated from the new tree.

Re-verified after the rebase (2 runs per arm, one ticket): strict census with LLAMA_INPUT_LAYER_ON_DEV=1 → 0 nodes off HRX; the same arm with the gate disabled → conflicting resident layout requests, 0 t/s; default path → the gate logs nothing and is unchanged (175.1 / 307.3 t/s). The decline log now names the registration and weight, so the mechanism is checkable rather than asserted. Full table in the body.

[bong-water-water-bong]

VERDICT: CHANGES at 27ecb2c3063693de74af0e2e9f3da2c2129b324a. One item left, and it's sequencing. The blocker from my CHANGES at 36fc187f is fixed, and the re-take checks out, but the pin is still the head of the OPEN fork PR #233.

Fixed and verified

• #231 is in the pin now: compare a85a6f34...092eb9db shows 0 commits missing, and at a85a6f34 the switch is #231's (ggml-hrx.cpp:163-166, enabled only for =1). So this bump no longer reverts #1339's mitigation. The body's "from" is now 092eb9dbc, and registry/architectures.json names a85a6f34.
• The re-take on the rebased base is real (~/wt/i1316-rv-1340, every run reports build b71-a85a6f342): default has 0 decline lines and runs at 175.1 / 307.3 t/s; dev-strict has 2 declines each and runs at 182.5 / 307.9; dev-nogate has 3 conflicting resident layout lines and 0.0 t/s. That matches the body's table, including "the gate does not fire" on the default path, which under #231's default is the device-copy path I asked to be re-measured. The decline lines name the registration and the weight, which is a good log line.

Remaining: compare a85a6f34...1bit/hrx-vulkan-patched = behind (pin 1 ahead, branch 0), and fork #233 is OPEN. Merge #233 onto the branch, then pin the commit it produces (re-pin if the squash rewrites the sha), as #1343 did with #231. Then this is READY with no further review.

[bong-water-water-bong]

VERDICT: READY at 27ecb2c30636 — with one title fix and one sequencing note, both of which the release lane will care about.

The #1343 trap does not bite here: the pin resolves. Same command, this head, rc 0:

$ git submodule update --init --depth 1 third_party/llama.cpp
   * branch  a85a6f34258e948fe1e1b9aa4cfa6fd19061ef86 -> FETCH_HEAD
   Submodule path 'third_party/llama.cpp': checked out 'a85a6f34258e948fe1e1b9aa4cfa6fd19061ef86'   rc 0
$ python3 tools/registry_build.py --check-pins
   registry/architectures.json records the committed pins                                             rc 0
registry: a85a6f34258e…   gitlink: a85a6f34258e   (agree — the #1267 trap satisfied)

1. The title names a sha that is not what is pinned, and it is not fetchable. The title says "bump llama.cpp to 37f89ee4";
the gitlink, the registry and the body all say a85a6f342 (the body even explains why: "the fork branch advanced while this was
in review … my commit is rebased onto that tip, so this bump goes 092eb9dbc → a85a6f342"). And 37f89ee4 is a divergent
pre-rebase commit: compare gives 1 ahead / 1 behind against the pin branch and 2 ahead / 1 behind against a85a6f34, a bare-sha
fetch refuses it, and its own registry patch is +49/−1 where the pinned commit's is +65/−1. So the title names a commit that
is in neither the branch nor this pin. Edit the title to a85a6f342 — this cycle has now produced three of these (a hash, an
address, and a title) and each one was quoted somewhere before it was caught.

2. The pin is fork PR #233's head, which is still OPEN (base 1bit/hrx-vulkan-patched, head a85a6f342), one commit on top of the
branch tip 092eb9dbc — so it is reachable today only because that PR ref is alive. That is the same shape as #1343, and it differs
only in that #1343's commit could not be fetched at all while this one can. If you would rather main not depend on an open PR's ref,
merge #233 first and the pin becomes the branch tip; I will re-verdict that head (it is a re-point, so the delta check is small).
Otherwise this is mergeable now and the re-point-after-#233 churn is a known cost.

3. "Was the proof run at that pin?" The body says "measured on the pin", and its pin is a85a6f342 — but because the title
names a different, divergent commit, that sentence is one ambiguity away from being unverifiable. One clause naming the sha the
measurement was taken at would close it, and it matters here more than usual: the #1316 fix is a dispatch/registry change
(dispatch-mul-mat.cpp +19/−1, dispatch-registry.cpp +65/−1), so a measurement at the pre-rebase commit would be a measurement of a
different patch.

Everything else is right: Fixes #1316 is the correct reference, only the pin line moves in the registry (counts stay 298/298), and
the fix is scoped to the case the issue names — a tied embedding no longer forcing one node onto the CPU per graph.

[github-actions]

Persistent review updated to latest commit a0c6d5f

[bong-water-water-bong]

VERDICT: READY at a0c6d5f6bfda5aa91aceebae750f0c6757095a1f. This clears the last item from my CHANGES at 27ecb2c3.

• The pin is the branch commit: third_party/llama.cpp = 67d8c42ec65ff6ec963b0d55d166ac888da2b438 = fork #233's merge commit, and compare 67d8c42e...1bit/hrx-vulkan-patched = identical (0/0). registry/architectures.json names the same sha. The engine diff is 2 files, +2/−2.
• The squash is what I verified: a direct tree diff of a85a6f34 (the head I checked at 27ecb2c3) against 67d8c42e is empty. So everything verified there carries over unchanged: #231's opt-in import is present (enabled only for =1), and the re-take on the rebased base holds (build a85a6f342; default path 0 declines; LLAMA_INPUT_LAYER_ON_DEV=1 0 nodes off HRX with 2 logged declines; gate off fails with the layout conflict).

Fixes #1316 is right: the report's symptom is explained as by-design CPU placement, and the opt-in override that was broken for tied embeddings now works.

[bong-water-water-bong]

#1344 re-pointed at the merged sha — re-verdict requested at a0c6d5f6b

Fork #233 is merged as 67d8c42ec65ff6ec963b0d55d166ac888da2b438, the tip of 1bit/hrx-vulkan-patched (ls-remote verified). This PR now:

• pins 092eb9dbc → 67d8c42ec in the gitlink and in registry/architectures.json (regenerated from the merged tree; counts 298/298);
• is retitled to 67d8c42ec;
• states, in the body, that the proof was measured at a85a6f342 and re-run at the merged sha — the merged diff is my patch byte-for-byte (+/- hash ca51bc22a8f40054` for both), and the merged commit is what the numbers are attached to.

Proof re-run at 67d8c42ec (2 runs per arm, one gpu-hold ticket, 0.6B, -dev HRX0 -ngl 99 -c 512):

arm	declines logged	census	gen t/s
LLAMA_INPUT_LAYER_ON_DEV=1 + GGML_HRX_STRICT=1	2	0 nodes off HRX	178.9 / 300.0
same, gate off (GGML_HRX_LAYOUT_CONFLICT_NO_PACK=0)	0	conflicting resident layout requests	0 / 0
default (…ON_DEV=0)	0 — gate not reached	unchanged	176.5 / 317.8

So the squash changed nothing observable: acceptance case green, the gate provably the fix, and the default path never reaching it (zero declines logged, which is mechanism-level rather than a timing comparison). Known-answer MUL_MAT 564 OK / 0 FAIL still stands on this tree.

[github-actions]

Persistent review updated to latest commit dbb8454