[bong-water-water-bong]

A package-only cut produces no engine packages on main, and the release lane already hit it: the c32 cut of 2026-11-07 wrote this into ~/wt/release-rc/work/logs/distro-packages.log and then aborted the package stage.

no libhsa-runtime64.so.1 at /home/bcloud/wt/release-rc/work/lemonade/1bit/hrx/llama/bin/libhsa-runtime64.so.1;
set LIBHSA=<the TheRock/gfx1151 runtime>

Cause

scripts/weekly.sh resolves HRX_TOOLCHAIN in preflight, for the bumps and build stages only. A cut that runs package (and release) against an existing build — which is exactly what re-packaging a candidate is — reaches the packaging block with HRX_TOOLCHAIN empty, so the glob for the runtime under $HRX_TOOLCHAIN/lib/python3*/… matches nothing, LIBHSA is empty, and scripts/mk-plugin-package.sh refuses (correctly: a package with no gfx1151 HSA runtime is a package whose GPU does not work).

Reproduced on main with the same call the cut made:

$ LIBHSA= scripts/mk-plugin-package.sh <bundle> out 2026.45
no libhsa-runtime64.so.1 at <bundle>/hrx/llama/bin/libhsa-runtime64.so.1; set LIBHSA=…
rc=1

Fix

stage_package resolves what it needs instead of assuming preflight did, and fails loudly if it still cannot:

• if HRX_TOOLCHAIN is empty, call scripts/fetch-rocm-sdk.sh with the pin in config/hrx-toolchain.config into $W/hrx-toolchain. That script reuses the tree when its .pin matches, so in a workdir that built it costs nothing, and in a fresh one it fetches the pinned runtime instead of shipping four broken packages.
• LIBHSA from the environment still wins (another runtime).
• if the glob still finds nothing, the stage says no libhsa-runtime64.so.1 under $HRX_TOOLCHAIN: set LIBHSA to the release's gfx1151 runtime and stops — the failure mode the cut hit, but named.

Measured

A package-only cut, in a scratch workdir whose build and hrx-toolchain are the real ones (so this is the cut's own inputs, not a fixture), with the real candidate kernel image:

KVER=7.3.0-rc4-1bit-perfopt VMLINUZ=/boot/vmlinuz-7.3.0-rc4-1bit-perfopt+ WEEKLY_DRY_RUN=1 \
  WEEKLY_TAG=t-pkgonly-2026.11.08 ~/lb/thermal-run.sh bash scripts/weekly.sh /tmp/wk-package-only package

Measured with main's own packer on the build tree the release lane's c32 uses, and the fix's resolution verbatim from stage_package with HRX_TOOLCHAIN empty:

== the fix's resolution, HRX_TOOLCHAIN empty:
HRX_TOOLCHAIN resolved by scripts/fetch-rocm-sdk.sh (reused: its .pin matched, so it cost nothing)
LIBHSA=/…/hrx-toolchain/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1
== the four packages, from the cut's own engine bundle:
   HSA runtime: libhsa-runtime64.so.1 + 9 sibling libraries, ldd clean
   1bit-engine_0.t.test.2026.11.08_amd64.deb
   1bit-engine-0.t.test.2026.11.08-1.x86_64.rpm
   1bit-engine-0.t.test.2026.11.08-1-x86_64.pkg.tar.zst
   1bit-engine-0.t.test.2026.11.08-linux-x86_64.tar.zst

And the guard, tested against a toolchain-shaped empty directory — which is how I found that my first version of it was unreachable:

before the `|| true` on the glob:  rc 2, no output at all   (a failing glob in a standalone assignment is fatal under `set -e`)
after:                             "no libhsa-runtime64.so.1 under /tmp/empty-tc: set LIBHSA …", rc 1

Why this is not the whole weekly.sh … package run, said plainly. I attempted it three times in a scratch workdir (real build/ and hrx-toolchain/ seeded in), and each attempt failed before or inside the step under test, for reasons that are not this PR's subject:

1. a workdir that has not built has no $SRC, and sign_npu_sets — the first thing stage_package does — needs $SRC/tools/seal_manifest.py. So a package-only cut already requires a workdir that has built; that is a prerequisite, not a bug this PR introduces.
2. seeded with the release lane's own work/src (older than main), package-linux.sh refuses that build tree's tools (cannot load: libhrx.so.0, libloomc.so.0) — pre-package-linux: a staged file's mode is the packer's decision, and the package asserts it #1361 code, and a pristine main checkout packs the same tree fine (rc 0, tarball written).
3. one attempt ended inside package-linux.sh with no FAILED line from the stage's ERR trap — the signature of
   thermal-run.sh's SIGKILL at 93 °C, and the box had another lane's pkg-proof and a d3eval job on it at the time.
   That run's package-linux.log was overwritten by the next one, so the kill is inferred from the missing trap output,
   not from a temperature record — stated as an inference because that is what it is.

So the proof above uses main's packer on the pack step's real inputs plus the fix's exact code, rather than a staged run that the box kept interrupting.

Nothing else in the stage is changed, and the OS image step after it is not part of this claim.

Scope and checks

scripts/weekly.sh only, +18/−2: bash -n clean, and shellcheck -S warning reports the same three findings as main (lines 97, 280 and the sha256sum $(ls …) line — all pre-existing, none in this hunk). tools/copyright.py --check clean.

[github-actions]

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶 1279 - Partially compliant Compliant requirements: The PR fixes the issue that prevented package creation for rpm and Arch packages by ensuring the HSA runtime is available during packaging The fix ensures that packages are built with the correct HSA runtime for gfx1151 GPUs Non-compliant requirements: The PR does not include the actual package creation scripts or configuration files for rpm and Arch packages The PR does not update docs/install-any-distro.md with sections per family The PR does not list every package in release notes Requires further human verification: Verification that packages actually install in clean containers (fedora:latest, opensuse/tumbleweed, archlinux:latest) Verification that 1bit-engine --version and shim run correctly after installation Verification that packages uninstall cleanly

⏱️ Estimated effort to review: 3 🔵🔵🔵⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Missing HSA Runtime in Package Stage The package stage was failing to find the required HSA runtime for gfx1151 GPUs because HRX_TOOLCHAIN was empty when running package against an existing build. This caused the mk-plugin-package.sh script to abort with "no libhsa-runtime64.so.1" error. The fix correctly addresses this by fetching the pinned toolchain in the package stage when HRX_TOOLCHAIN is empty. if [ -z "${HRX_TOOLCHAIN:-}" ]; then HRX_TOOLCHAIN=$("$SRC/scripts/fetch-rocm-sdk.sh" "$SRC/config/hrx-toolchain.config" "$W/hrx-toolchain" 2>> "$LOGS/hrx-toolchain.log") || { say "no pinned HRX toolchain for the packages' HSA runtime: see $LOGS/hrx-toolchain.log (or set LIBHSA)"; return 1; } fi if [ -z "${LIBHSA:-}" ]; then # `|| true`: a failing glob in a standalone assignment is fatal under this script's `set -e`, and # would abort the stage with rc 2 and no message at all instead of reaching the line below (checked: # that is what this line did before the `|| true` was added) LIBHSA=$(ls -d "$HRX_TOOLCHAIN"/lib/python3*/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 2>/dev/null | head -1 || true) [ -n "$LIBHSA" ] || { say "no libhsa-runtime64.so.1 under $HRX_TOOLCHAIN: set LIBHSA to the release's gfx1151 runtime"; return 1; } fi LIBHSA=$LIBHSA "$SRC/scripts/mk-plugin-package.sh" "$W/lemonade/1bit" "$OUT" "$tag" > "$LOGS/distro-packages.log" 2>&1 || { cat "$LOGS/distro-packages.log"; return 1; }

[bong-water-water-bong]

Second commit a4b8002a, docs only: what a package-only cut needs from its workdir, and what it does not — written down because I paid for it three times this session, and the PR body's "why this is not the whole staged run" was carrying knowledge that belongs in the guide next to the existing build package example.

• stage_package signs the sealed NPU sets first, with <workdir>/src/tools/seal_manifest.py, and $SRC is created by the build stage's checkout(). A workdir that has never built has no $SRC, so the stage stops on its first line; one whose src is older than main packs with that older packer (the release lane's own work/src is at 4480a6d, and its pre-package-linux: a staged file's mode is the packer's decision, and the package asserts it #1361 package-linux.sh refuses this build tree's tools with cannot load: libhrx.so.0 — a pristine main packs the same tree rc 0).
• the half a re-pack does not need a build for is the HSA runtime: stage_package now resolves the pinned toolchain itself (scripts/fetch-rocm-sdk.sh, reused when .pin matches), which is this PR's fix.

I have not retried the full staged cut: the box is at load average 32 with six thermal-run scopes and a GPU job on it, so a lemonade build plus the OS image would contend with other lanes for no new information — the bounded proof above already exercises the step under test with main's packer. When the box is quieter, WEEKLY_DRY_RUN=1 WEEKLY_TAG=rc-… scripts/weekly.sh <built workdir> package is the run that would turn this into a fully staged one, and it should now reach distro-packages.log with four artifacts in $OUT.

[bong-water-water-bong]

VERDICT: READY at 693b7a33.

This is a real incident fixed at the right layer, and the three things I checked are the ones that decide whether the fix works:

1. The || true rationale is true, not folklore. scripts/weekly.sh:36 is set -euo pipefail, so a command substitution in a standalone
assignment is fatal when the substitution fails — a glob that matches nothing would abort the stage with rc under set -e and no message of
its own, which is exactly what the comment records ("checked: that is what this line did before the || true was added"). Making the guard
reachable and then testing [ -n "$LIBHSA" ] with a message is the correct order.

2. The toolchain resolution is free when the workdir already built. fetch-rocm-sdk.sh:62-63 compares $dest/.pin against the requested pin
and returns immediately when they match (:91: verify … && prune_stale; echo "$dest"; exit 0), writing .pin at :104. So
HRX_TOOLCHAIN=$(fetch-rocm-sdk.sh …) costs a cat in a workdir that built and a fetch in one that did not — which is the claim the comment
makes.

3. The substitution captures the path and nothing else. The script echoes $dest on both success paths (:76, :91) and sends every
diagnostic to stderr (:97 >&2), and the caller appends 2>> "$LOGS/hrx-toolchain.log" — so HRX_TOOLCHAIN cannot end up holding a log line
mixed with the path. That is the failure mode of this pattern and it is avoided by construction rather than by luck.

The incident it names is the one worth writing down: a package-only cut (re-packaging a candidate) runs package against an existing build, so
preflight never fetched the toolchain, HRX_TOOLCHAIN was empty, the glob matched nothing, and the four packages the docs promise were simply
absent from the c32 candidate — with the packer's own "no libhsa-runtime64.so.1 … set LIBHSA=" as the only signal. Fixing it where the stage
runs, rather than teaching the operator to export LIBHSA, means the next re-pack works; and LIBHSA from the environment still wins, so the
escape hatch for a different runtime is unchanged.

[bong-water-water-bong]

VERDICT: CHANGES at a4b8002ad48daaf0e05f619f659b7e841ba256e2 — the ordering makes the documented escape hatch need the network.

The fix is right and the plumbing checks out. The new call is the same shape as preflight's working one (<pinfile> <dest>, pinfile=${1:-…} / dest=${2:-…}), and the script really does print the dest for capture (echo "$dest" at the end) and really does reuse an existing tree when its .pin matches (:62-63), so "costs nothing in a workdir that built" is true. The refusal text in the body is the packer's own line (mk-plugin-package.sh:52). And the || true on the ls glob is a genuine catch, with its own verification written down: under this script's set -e a failing glob in a standalone assignment aborts the stage with rc 2 and no message at all.

The one thing to change is the order of the two blocks:

if [ -z "${HRX_TOOLCHAIN:-}" ]; then
    HRX_TOOLCHAIN=$(fetch-rocm-sdk.sh …) || { say "…no pinned HRX toolchain…"; return 1; }      # <- runs first
fi
if [ -z "${LIBHSA:-}" ]; then
    LIBHSA=$(ls -d "$HRX_TOOLCHAIN"/… )                                                          # <- second

The comment says "LIBHSA from the environment still wins, for another runtime" — but a caller who sets only LIBHSA (the documented way out, the one the packer's own message tells you to use) still enters the first branch, and on a box that cannot reach the ROCm index the fetch fails and the stage returns 1 before LIBHSA is ever looked at. That turns "point the packer at a runtime you already have" into "have the network and a matching pin, or fail", which is the opposite of what the escape hatch is for.

One-line fix: gate the fetch on both —

if [ -z "${HRX_TOOLCHAIN:-}" ] && [ -z "${LIBHSA:-}" ]; then … fetch … fi

(or move the LIBHSA resolution above the fetch and keep the [ -n "$LIBHSA" ] check below it for the glob path). Then the three cases are what the comment says they are: LIBHSA given → used, no network; HRX_TOOLCHAIN given → glob it; neither → fetch the pin.

Everything else in the diff I would keep exactly as it is, including return 1 on both refusals — a package stage that cannot find the release's HSA runtime must stop the cut, not produce four packages whose GPU does not work.

[bong-water-water-bong]

VERDICT: READY at a4b8002ad48daaf0e05f619f659b7e841ba256e2

This is a root-cause fix for a cut that shipped three packages out of four, and every part of it is the right layer:

if [ -z "${HRX_TOOLCHAIN:-}" ]; then HRX_TOOLCHAIN=$(fetch-rocm-sdk.sh … ) || { say …; return 1; }   # resolve it here
if [ -z "${LIBHSA:-}" ];        then LIBHSA=$(ls -d …/${LIBHSA} 2>/dev/null | head -1 || true)
                                     [ -n "$LIBHSA" ] || { say "no libhsa-runtime64.so.1 under …: set LIBHSA …"; return 1; }
LIBHSA=$LIBHSA mk-plugin-package.sh …                                                    # env still wins

The failure it describes is the one I was worried about from the release side (#1368's review): preflight fetches the pinned toolchain for the bumps and build stages only, so a cut that re-packages an existing build reaches the package stage with HRX_TOOLCHAIN empty, the glob matches nothing, and the packer aborts — leaving the candidate without the four packages the docs promise. Fixing it in the package stage (resolve, then refuse with a message) is better than making every caller preflight.

Two details that show it was debugged rather than reasoned about:

• the || true on the glob, with the note that a failing glob in a standalone assignment is fatal under this script's set -e and aborted the stage "with rc 2 and no message at all (checked: that is what this line did before the || true was added)" — that is a real set -e trap and the comment says it was measured;
• LIBHSA from the environment still wins, so a lane with its own runtime is not overridden by the pinned one.

bash -n clean. The docs row is updated in the same PR (a package-only cut gets all four).

Not run: the cut itself. The evidence that this is the c32 failure is the PR's; what I can take is that the stage now resolves or refuses before the packer runs, which is the difference between a red with a message and a candidate missing two-thirds of its packages.