diff --git a/docs/hrx.md b/docs/hrx.md index b6dffedb0..b877cb368 100644 --- a/docs/hrx.md +++ b/docs/hrx.md @@ -162,6 +162,14 @@ were observed. 5.4K-token prompt in a new process 4.46 -> 3.43 s, first short prompt 0.47 -> 0.22 s, with identical output. `GGML_HRX_JIT_CACHE=0` turns it off; it is off while a Loom sanitizer is set. Deleting the directory is always safe. +- **Fused residual scale (llama.cpp #35).** ZAYA scales both residual branches twice per layer, + `(x + bx) * sx + (r + br) * sr`, which ran as up to nine ADD / MUL dispatches per layer. Each side + now runs as one dispatch (`common.res_scale_pair.*`, `ops/res_scale_pair_f32.loom`): 3,366 -> 3,165 + dispatches per token, bitwise the same output, ZAYA1-8B HRX0 decode 67.0 -> ~69.6 tok/s and + pp512 2,109 -> 2,183. `GGML_HRX_DISABLE_DISPATCH=res_scale_pair` turns it off. At decode most of + a ZAYA token is small dispatches and the ~1.8 us gaps between them, not matmuls; the profiler + (`HRX_PROFILE_MODE=dispatch`) shows which ones. Profile through `llama-server`: `llama-bench` + under the profiler fails. ### Known issues on `HRX0` diff --git a/registry/architectures.json b/registry/architectures.json index 3c2a20c5a..1c404230a 100644 --- a/registry/architectures.json +++ b/registry/architectures.json @@ -2,7 +2,7 @@ "about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.", "sources": { "llama.cpp (vulkan)": "cecf3ee01d9d99378e98bfea95f51cac714b04b8", - "llama.cpp (hrx)": "b408831d6beffabfb054f69748ad57338f054cce", + "llama.cpp (hrx)": "45c17708344627b0731ff28d4075da65a3bfa6ff", "zinc": "29bc350ac4cbf9f110ec628b9e177ea04ac816fb" }, "counts": { diff --git a/third_party/llama.cpp b/third_party/llama.cpp index b408831d6..45c177083 160000 --- a/third_party/llama.cpp +++ b/third_party/llama.cpp @@ -1 +1 @@ -Subproject commit b408831d6beffabfb054f69748ad57338f054cce +Subproject commit 45c17708344627b0731ff28d4075da65a3bfa6ff