From 693b7a3376debd5ddc9d6a45399ce69f44ea8bcb Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 02:31:08 -0400 Subject: [PATCH 1/2] weekly: a package-only cut gets its HSA runtime, so its four packages exist (Refs #1279) The c32 cut of 2026-11-07 wrote this into its own log and then aborted the package stage: no libhsa-runtime64.so.1 at .../work/lemonade/1bit/hrx/llama/bin/libhsa-runtime64.so.1; set LIBHSA= Cause, and it is in the wiring this lane added: preflight resolves HRX_TOOLCHAIN for the `bumps` and `build` stages only. A cut that runs `package` (and `release`) against an existing build - which is what re-packing a candidate is - reaches the packaging block with HRX_TOOLCHAIN empty, so the glob under $HRX_TOOLCHAIN/lib/python3*/... matches nothing, LIBHSA is empty, and mk-plugin-package.sh refuses. It refuses correctly (a package with no gfx1151 runtime is a package whose GPU does not work); the wiring was what assumed a stage that had not run. Reproduced on main with the cut's own call: LIBHSA= scripts/mk-plugin-package.sh out 2026.45 no libhsa-runtime64.so.1 at /hrx/llama/bin/libhsa-runtime64.so.1; set LIBHSA=... rc=1 Fixed in stage_package, where the need is: if HRX_TOOLCHAIN is empty it calls scripts/fetch-rocm-sdk.sh with the pin in config/hrx-toolchain.config into $W/hrx-toolchain - which reuses the tree when its .pin matches, so a workdir that built pays nothing and a fresh one fetches the pinned runtime instead of shipping four packages whose GPU cannot work. LIBHSA from the environment still wins. The guard is `|| true` on the glob for a measured reason: a failing glob in a STANDALONE assignment is fatal under this script's `set -e`, so the first version aborted with rc 2 and no message at all instead of reaching the "no libhsa-runtime64.so.1 under $HRX_TOOLCHAIN: set LIBHSA ..." line. Found by testing the guard itself against a toolchain-shaped empty directory: before `|| true`: rc 2, no output after: "no libhsa-runtime64.so.1 under /tmp/empty-tc: set LIBHSA ...", rc 1 Measured, with main's own packer on the build tree the release lane's c32 uses, and the fix's resolution verbatim from stage_package with HRX_TOOLCHAIN empty: HRX_TOOLCHAIN resolved by scripts/fetch-rocm-sdk.sh (reused: its .pin matched) LIBHSA=/.../hrx-toolchain/lib/python3.14/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 four packages from the cut's own engine bundle, HSA runtime + 9 siblings, ldd clean: 1bit-engine_0.t.test.2026.11.08_amd64.deb 1bit-engine-0.t.test.2026.11.08-1.x86_64.rpm 1bit-engine-0.t.test.2026.11.08-1-x86_64.pkg.tar.zst 1bit-engine-0.t.test.2026.11.08-linux-x86_64.tar.zst Two things about the full cut that this measurement had to work around, both worth knowing rather than rediscovering: `weekly.sh package` on a workdir that has not built cannot work at all, because sign_npu_sets (which runs first in the stage) needs $SRC/tools/seal_manifest.py and only the build stage creates $SRC; and the release lane's own work/src is older than main, where package-linux.sh refuses this build tree's tools (pre-#1361). Neither is this PR's subject. checks: bash -n clean; shellcheck -S warning reports the same three findings as main (97, 280 and the sha256sum $(ls ...) line, all pre-existing, none in this hunk); tools/copyright.py --check rc 0. --- scripts/weekly.sh | 23 +++++++++++++++++++++-- 1 file changed, 21 insertions(+), 2 deletions(-) diff --git a/scripts/weekly.sh b/scripts/weekly.sh index e018a05e..d9c7e000 100755 --- a/scripts/weekly.sh +++ b/scripts/weekly.sh @@ -553,8 +553,27 @@ stage_package() { # the same in all four (scripts/pack-distro.sh). LIBHSA is the release's gfx1151 runtime out of the # pinned toolchain: the packer refuses a runtime that would register no device, and the engine # bundle carries none (HRX dlopens it). - LIBHSA=$(ls -d "$HRX_TOOLCHAIN"/lib/python3*/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 2>/dev/null | head -1) \ - "$SRC/scripts/mk-plugin-package.sh" "$W/lemonade/1bit" "$OUT" "$tag" > "$LOGS/distro-packages.log" 2>&1 || + # The four packages need the release's gfx1151 HSA runtime and the engine bundle does not carry one (HRX + # dlopens it). preflight fetches the pinned toolchain for the bumps and build stages only, so a cut that + # runs package (and release) against an existing build - which is what re-packaging a candidate is - gets + # here with HRX_TOOLCHAIN empty, the glob below matches nothing, and the packer aborts with "no + # libhsa-runtime64.so.1 ... set LIBHSA=": the c32 cut of 2026-11-07 did exactly that, and the four + # packages the docs promise were missing from the candidate. Resolve it here instead: fetch-rocm-sdk.sh + # reuses the tree when its .pin matches (so it costs nothing in a workdir that built) and fetches the + # pinned runtime in one that did not. LIBHSA from the environment still wins, for another runtime. + if [ -z "${HRX_TOOLCHAIN:-}" ]; then + HRX_TOOLCHAIN=$("$SRC/scripts/fetch-rocm-sdk.sh" "$SRC/config/hrx-toolchain.config" "$W/hrx-toolchain" 2>> "$LOGS/hrx-toolchain.log") || + { say "no pinned HRX toolchain for the packages' HSA runtime: see $LOGS/hrx-toolchain.log (or set LIBHSA)"; return 1; } + fi + if [ -z "${LIBHSA:-}" ]; then + # `|| true`: a failing glob in a standalone assignment is fatal under this script's `set -e`, and + # would abort the stage with rc 2 and no message at all instead of reaching the line below (checked: + # that is what this line did before the `|| true` was added) + LIBHSA=$(ls -d "$HRX_TOOLCHAIN"/lib/python3*/site-packages/_rocm_sdk_core/lib/libhsa-runtime64.so.1 2>/dev/null | head -1 || true) + [ -n "$LIBHSA" ] || + { say "no libhsa-runtime64.so.1 under $HRX_TOOLCHAIN: set LIBHSA to the release's gfx1151 runtime"; return 1; } + fi + LIBHSA=$LIBHSA "$SRC/scripts/mk-plugin-package.sh" "$W/lemonade/1bit" "$OUT" "$tag" > "$LOGS/distro-packages.log" 2>&1 || { cat "$LOGS/distro-packages.log"; return 1; } say "package $tag: proof, one clean container per family" # Nothing is published that nobody installed: every family is installed in a throwaway container of From a4b8002ad48daaf0e05f619f659b7e841ba256e2 Mon Sep 17 00:00:00 2001 From: bong-water-water-bong Date: Sun, 8 Nov 2026 03:01:12 -0400 Subject: [PATCH 2/2] docs: a package-only cut needs a workdir that has built, and what it does not need (Refs #1279) Found by paying for it: three attempts at `weekly.sh package` on a scratch workdir, each stopped by a prerequisite nobody had written down. - stage_package signs the sealed NPU sets *first*, with /src/tools/seal_manifest.py, and $SRC is created by the build stage's checkout. A workdir that has never built has no $SRC at all, so the stage stops on its first line; one whose src is older than main packs with that older packer (the release lane's own work/src is at 4480a6d, and its pre-#1361 package-linux.sh refuses this build tree's tools with 'cannot load: libhrx.so.0', while a pristine main packs the same tree rc 0). - the HSA runtime is the half a re-pack does NOT need a build for: stage_package resolves the pinned toolchain itself (scripts/fetch-rocm-sdk.sh, reusing the tree when its .pin matches). It sits next to the existing 'exercise only the private builds and the packaging' example, which already runs build before package for this reason. Measured, not inferred: each failure line above is a log or a reproduction from that session, except the one attempt that ended with no ERR-trap output, which the PR now describes as an inferred thermal kill. docs only, +10 lines; copyright --check rc 0. --- docs/releases.md | 10 ++++++++++ 1 file changed, 10 insertions(+) diff --git a/docs/releases.md b/docs/releases.md index 367c9193..27b148db 100644 --- a/docs/releases.md +++ b/docs/releases.md @@ -474,6 +474,16 @@ To exercise only the private builds and the packaging without releasing, on Stri WEEKLY_DRY_RUN=1 WEEKLY_TAG=rehearsal-private scripts/weekly.sh ~/.cache/1bit-weekly build package ``` +That example runs `build` before `package` for a reason, and it is not only the build: **a `package`-only run +needs a workdir that has already built.** `stage_package` starts by signing the sealed NPU sets, and it signs +them with `/src/tools/seal_manifest.py` — the source checkout that the `build` stage's `checkout()` +creates. On a workdir that has never built, `weekly.sh package` therefore stops on its first line +with `can't open file '…/src/tools/seal_manifest.py'`, and on one whose `src` is older than `main` it packs +with that older packer (measured 2026-11-08: three attempts, each stopped by one of those two, plus a `src` +whose pre-#1361 `package-linux.sh` refused the build tree's tools). What a re-pack does *not* need a build +for is the HSA runtime: `stage_package` resolves the pinned toolchain itself with +`scripts/fetch-rocm-sdk.sh` (`config/hrx-toolchain.config`), which reuses the tree when its `.pin` matches. + ## The release run `scripts/weekly.sh ` on Strix Halo, in stages: