Repository navigation
hrx: cache Loom JIT results on disk across processes (first-request warm-up) - #33
Merged
bong-water-water-bong merged 1 commit intoSep 29, 2026
Conversation
HRX compiles each kernel specialization on first dispatch and keeps it only in
the process, so every server start pays the compiles again: on ZAYA1-8B each
new prompt-length remainder compiles ~24 specializations (~1 s). The new
runtime/loom-jit-disk-cache.{h,cpp} (ours) stores each result (HSACO, manifest,
launch config) under $GGML_HRX_JIT_CACHE_DIR (default ~/.cache/1bit/hrx-jit),
keyed on every compile input (source and dependency bytes, config bindings,
workload values, target) plus the path/size/mtime of the library holding the
compiler, so a rebuilt or updated HRX never reuses old code. Writes are atomic
(temp file + rename); a damaged entry is recompiled. GGML_HRX_JIT_CACHE=0
disables it; it is off while a Loom sanitizer is set. Hookups in
loom-kernel-jit.cpp: load before compile, store after, record the target.
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
merged commit Sep 29, 2026
b408831
into
1bit/hrx-vulkan-patched
10 of 24 checks passed
bong-water-water-bong
pushed a commit
to 1bit-MONSTER/engine
that referenced
this pull request
Sep 29, 2026
…uest warm-up) Brings 1bit-MONSTER/llama.cpp#33: each compiled kernel specialization is kept under ~/.cache/1bit/hrx-jit (GGML_HRX_JIT_CACHE_DIR), keyed on every compile input and the compiler library's identity, so a new server process skips the compiles. ZAYA1-8B on HRX0, new process: first 5.4K-token prompt 4.46 -> 3.43 s, a new 3K prompt length 1.84 -> 1.55 s, first short prompt 0.47 -> 0.22 s; outputs identical with the cache on and off. GGML_HRX_JIT_CACHE=0 disables it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
bong-water-water-bong
added a commit
to 1bit-MONSTER/engine
that referenced
this pull request
Sep 29, 2026
…uest warm-up) (#219) Brings 1bit-MONSTER/llama.cpp#33: each compiled kernel specialization is kept under ~/.cache/1bit/hrx-jit (GGML_HRX_JIT_CACHE_DIR), keyed on every compile input and the compiler library's identity, so a new server process skips the compiles. ZAYA1-8B on HRX0, new process: first 5.4K-token prompt 4.46 -> 3.43 s, a new 3K prompt length 1.84 -> 1.55 s, first short prompt 0.47 -> 0.22 s; outputs identical with the cache on and off. GGML_HRX_JIT_CACHE=0 disables it. Co-authored-by: bong-water-water-bong <bong-water-water-bong@1bit.gg> Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Engine RFC #213, gate 4 (first-request latency).
Problem: HRX compiles every kernel specialization on first dispatch and keeps it only in memory, so each server start pays the compiles again. A dispatch profile shows that the compile parameters bake in the ubatch token count. So each new prompt-length remainder compiles a fresh set: 2,048 vs 2,112 tokens gave 24 new specializations on ZAYA1-8B, about 1 s the first time.
Change:
runtime/loom-jit-disk-cache.{h,cpp}(ours) stores each Loom JIT result (HSACO, manifest JSON, launch config) under$GGML_HRX_JIT_CACHE_DIR(default~/.cache/1bit/hrx-jit).GGML_HRX_JIT_CACHE=0disables it, and it is off while a Loom sanitizer is set.loom-kernel-jit.cppgains three hookups (load before compile, store after, record the target), plus two CMake lines.Measured on ZAYA1-8B, HRX0, through llama-server, each time in a new process:
What remains on the first request (~0.6 s at 5.4K) is other one-time setup, not compilation. A full cache for this run is 486 entries, 5.9 MB.
Correctness: greedy decoding with top-5 log-probs over 6 prompts × 48 tokens is identical with the cache on and off: 288/288 tokens, max |Δ logprob| 0.0000.
🤖 Generated with Claude Code