Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions docs/hrx.md
Original file line number Diff line number Diff line change
Expand Up @@ -162,6 +162,14 @@ were observed.
5.4K-token prompt in a new process 4.46 -> 3.43 s, first short prompt 0.47 -> 0.22 s, with
identical output. `GGML_HRX_JIT_CACHE=0` turns it off; it is off while a Loom sanitizer is set.
Deleting the directory is always safe.
- **Fused residual scale (llama.cpp #35).** ZAYA scales both residual branches twice per layer,
`(x + bx) * sx + (r + br) * sr`, which ran as up to nine ADD / MUL dispatches per layer. Each side
now runs as one dispatch (`common.res_scale_pair.*`, `ops/res_scale_pair_f32.loom`): 3,366 -> 3,165
dispatches per token, bitwise the same output, ZAYA1-8B HRX0 decode 67.0 -> ~69.6 tok/s and
pp512 2,109 -> 2,183. `GGML_HRX_DISABLE_DISPATCH=res_scale_pair` turns it off. At decode most of
a ZAYA token is small dispatches and the ~1.8 us gaps between them, not matmuls; the profiler
(`HRX_PROFILE_MODE=dispatch`) shows which ones. Profile through `llama-server`: `llama-bench`
under the profiler fails.

### Known issues on `HRX0`

Expand Down
2 changes: 1 addition & 1 deletion registry/architectures.json
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,7 @@
"about": "HF architecture -> GGUF architecture and the backends whose code accepts it. Generated by tools/registry_build.py from the pinned sources; do not edit.",
"sources": {
"llama.cpp (vulkan)": "cecf3ee01d9d99378e98bfea95f51cac714b04b8",
"llama.cpp (hrx)": "b408831d6beffabfb054f69748ad57338f054cce",
"llama.cpp (hrx)": "45c17708344627b0731ff28d4075da65a3bfa6ff",
"zinc": "29bc350ac4cbf9f110ec628b9e177ea04ac816fb"
},
"counts": {
Expand Down
Loading