Skip to content

prepare-stick: check the space before staging, and report the filesystem's size, not the partition's - #1388

Merged
bong-water-water-bong merged 5 commits into
mainfrom
os/v3-stick-preflight
Nov 8, 2026
Merged

bong-water-water-bong merged 5 commits into
mainfrom
os/v3-stick-preflight

Conversation

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator

Fixes #1374

Root cause

os/mini/prepare-stick.sh staged models with cp per file and only learned that the data filesystem was full from
cp's own failure — and a cp that fails half way leaves a truncated model behind. On c32 the partition had grown to
58.1 GiB while the filesystem stayed 7.8 GiB, so 6.6 of 7.9 GB landed; the partial tree could not be shown by du -sb,
was found by byte-comparing against the source, and was freed by hand (e2fsck -f → resize2fs → clear → restage). This
is the second occurrence.

The change (one file, +65/−6)

  1. A preflight before anything is written. The bytes the models need are compared with df's available bytes on the
    mounted data partition; a shortfall dies with both numbers and where to look (grow/grown lines, sfdisk -d for the
    partition vs df -h for the filesystem). Only the growth is counted — cp -f truncates a file that is already
    there — so a re-run, and the restage path this issue needed, are not refused for space they already occupy.
  2. grow_data's silent fallback stays, exactly as os: the live stick's /data is 2 GB (351 MB free after boot) - no catalogue model can be pulled on the live appliance #706 wants (a failed grow leaves a working 2 GiB stick). What
    changed is that staging now notices instead of copying into a filesystem that cannot hold the file.
  3. resize2fs is retried once after e2fsck -f -y — an ext4 needing journal recovery makes resize2fs refuse, and
    that hand-run fsck was the board's workaround — and a second refusal dies saying staging into it would truncate.
  4. The numbers come from the filesystem: grown reads both sizes with dumpe2fs -h, and the data line says
    "of partition". Quoting the partition table as the filesystem's size is what made the c32 failure read as a success.

The script's header documents the space rule (rule 5: docs with the behaviour).

Measured — file-backed sticks, no stick, no device

Same 48 MiB model into an ext4 with 24 MiB free (the c32 shape: an image file, so grow_data never runs):

result
before (main) cp: error writing …: No space left on device → dies, and leaves 26 214 400 of 50 331 648 bytes — a truncated big-model.gguf — on the medium
after (this branch) prepare-stick: refusing to stage: the data filesystem has 24727552 bytes free and the models need 51380224 … (engine#1374) — exit 1, and models/ is not created: nothing to prove, nothing to clean up

The healthy paths are unchanged, which is the part a refusal must not break:

case result
a 20 MiB model into that same stick stages, every assertion PASS, exit 0
re-run of that same model with 3.6 MiB free (growth 0) not refused, prepared and proved, exit 0
a 48 MiB model on that same nearly-full stick refused with both numbers, exit 1
whole-disk loop target (so grow_data runs) grown /dev/loop17p2 now holds 0.2 GiB of ext4 (was 0.0 GiB; both read from the filesystem, not the partition), model staged, exit 0
resize2fs stub that refuses its first call grown resize2fs refused (…) : e2fsck -f -y, then one retry → e2fsck -f -y logged, resize2fs called twice, exit 0
resize2fs stub that refuses twice resize2fs failed on … twice, once after e2fsck -f -y: … staging into it would truncate …, exit 1, nothing staged

Two honesty notes on the measurement: the retry row uses a stub for resize2fs (I could not make a real ext4 refuse
here — debugfs ssv state 1 did not), so that row proves the branch's logic, not a real corrupt filesystem; and my first
attempt at the reproduction was invalid because truncate -s 48M makes a sparse source, which cp copies without
allocating — the numbers above use /dev/urandom data (blocks 98304, non-sparse), which is what makes cp hit ENOSPC.

sh -n and bash -n clean, tools/copyright.py --check rc 0. Every loop device and mount from the measurement released.

Not in this PR

No gate change (the release gates keep byte-verifying the staged tree — that is how this was caught, and they should not be
the thing that discovers a truncated model). No change to grow_data's fallback, to the GPT handling, or to what is
staged. No image build, no board run, no device ticket; :18097 and npu-serve-keeper untouched.

…tem's size, not the partition's

c32's chain died ENOSPC after a successful write/readback: the data partition had grown to
58.1 GiB and the filesystem was still 7.8 GiB, so 6.6 of 7.9 GB of models landed and
sd_turbo.safetensors was truncated. The partial tree could not be proved by du -sb, so it was
found by byte-comparing against the source and freed by hand: e2fsck -f, resize2fs, clear,
restage (engine#1374, second occurrence).

Four things in this script, all of them the issue's:

1. The staging loops now run a preflight: the bytes the models need - only the GROWTH over a
   file cp -f would truncate, so a re-run or the restage path is not refused for space it
   already holds - are compared with df's available bytes on the mounted data partition, and a
   shortfall dies with both numbers and where to look (the 'grow'/'grown' lines, sfdisk -d vs
   df -h). Nothing is written, so there is nothing to prove or clean up afterwards.
2. grow_data's silent fallback stays, exactly as #706 wants it: a failed grow still leaves a
   working 2 GiB stick. What changed is that the staging now notices instead of copying into a
   filesystem that cannot hold the file.
3. A resize2fs that refuses is retried ONCE after e2fsck -f -y (an ext4 needing journal
   recovery makes resize2fs refuse, and the board's workaround was exactly that by hand), and
   a second refusal dies saying staging into it would truncate.
4. The 'grown' line reads both numbers from the filesystem (dumpe2fs -h) and the 'data' line
   says "of partition": quoting the partition table as the filesystem's size is what made the
   c32 failure read as a success.

Measured on file-backed sticks (no stick, no device), same 48 MiB model into a 24 MiB-free ext4:

    before   cp: No space left on device -> dies, and leaves 26214400 of 50331648 bytes
             (a TRUNCATED big-model.gguf) on the medium
    after    prepare-stick: refusing to stage: the data filesystem has 24727552 bytes free and
             the models need 51380224 ... (engine#1374)  - exit 1, models/ not created

Healthy path unchanged: a 20 MiB model into the same stick stages and passes every assertion
(exit 0); a re-run of that same model with 3.6 MiB free is NOT refused (growth 0, exit 0) and a
bigger model on that stick is. A whole-disk loop target exercises the grow: 'grown ...
0.2 GiB of ext4 (was 0.0 GiB; both read from the filesystem, not the partition)'. The retry
branch was measured with a resize2fs stub that refuses its first call: e2fsck -f -y was called,
resize2fs ran twice, exit 0; a stub that refuses twice dies before any staging.

Fixes #1374

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at d04b259d25f2a980796b2362657cd6dbaad0fe97.

The fix is a refusal path, so the properties that matter are "does the number match the units" and "can it refuse something that would have worked". Both are right.

Units and accounting, line by line

  • avail= is bytes, compared against a need built from stat -Lc %s (bytes) and du -sb (bytes) — the classic 1 KiB-vs-byte mismatch that would make this refuse a stick with 4× the space is not there.
  • Only growth is counted: [ "" -gt "" ] && need=0, with old read from the destination path cp -f would truncate. That is the half that makes the recovery path work at all — the restage after c32's failure would otherwise be refused for space the tree already holds.
  • The NPU-directory row accounts for the rm -rf (its old bytes come back to the filesystem), which is the other place a naive "sum the sources" would over-refuse.
  • A df that cannot answer skips rather than refuses ([ -n "" ]), and the comment names why: an old busybox has no --output. Failing open here is right — cp is still the backstop, and a false refusal on a tool gap would be a new way to lose a stick.
  • 1 MiB of headroom for the entries and the conf/key writes, and the refusal message names the two commands that separate the two numbers (sfdisk -d for the partition, df -h for the filesystem), which is the fault this issue is about.

Ordering and mode: the check sits after the mount and before any write_key/cp, so a refusal writes nothing at all ("nothing to prove and nothing to clean up"), and it is skipped when CHECK=0 because that mode stages nothing.

The other half of the fix is the more valuable one in hindsight: grown now reads both numbers from the filesystem (dumpe2fs -h, block count × block size) instead of from the partition table, and the data line says "of partition" — because quoting the partition's 58.1 GiB while the ext4 was 7.8 GiB is exactly what made c32's failure read as a success. Retrying resize2fs once after e2fsck -f -y (the board's own workaround, done at 2am by hand) is the same instinct.

One suggestion, not a blocker: the subtle half — "only growth counts" — is the part with no test, and it is testable without a device (a fake `` tree plus a df seam would pin it). I would not hold this PR for that: the change turns a truncating failure into a refusal, so landing it un-tested cannot make a stick worse than it is today.

Micro-edge, for completeness: if stat -Lc %s failed on a model that exists but is unreadable, new would be empty and [ "" -gt 0 ] would be a bash arithmetic error rather than this script's die message. The [ -s "" ] || continue guard covers the realistic case (a missing model), so this is a note rather than an ask.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: NOT READY at d04b259 — for the base, not the work. The prepare-stick change is good and says more than the body claims; the
problem is that the branch predates a merge and quietly undoes it.

The blocker: it reverts #1383, which merged 34 minutes ago

merge-base --is-ancestor origin/main d04b259      ->  NO (stale base)
git diff origin/main d04b259 --stat
   docs/npu.md              | 17 ++++--------      <- #1383's merged text, reverted
   os/mini/prepare-stick.sh | 71 ++++++++++++++++

#1383 merged at 01:21:33Z; this branch was cut before it, so its docs/npu.md still holds the older "The two patches every build
of ours runs" wording and merging it would take the merged sentence — nine patches, seven applied, 0006/0007 carried and not installed —
back off main. That is why the body's "one file, +65/−6" and the diff disagree: the extra file is the revert, not your change. A rebase
onto current main should leave prepare-stick.sh alone and drop docs/npu.md entirely. Fourth stale-base collision tonight — the
two-line check catches all of them before review, and this one is the cheapest kind: no code of yours is wrong, one file is extra.

The work itself: verified, and it is a superset of the open #1375

Checks that do not need root, run at this head: sh -n rc 0, bash -n rc 0, copyright --check rc 0; the fit check sits at :280,
after the mount at :271, so df is reading the data filesystem and the old sizes are real. And your two headline cases
reproduce from the code as arithmetic, not from the prose:

48 MiB model, nothing there, 24 MiB free :  need = 48 + 1 MiB = 51380224 > 25165824  -> REFUSE
re-run, 45.6 MiB already on the medium, 3.6 MiB free:  need = 2.4 + 1 MiB = 3565158 < 3774873  -> pass

The growth-only accounting is right for both kinds of payload: stat -Lc %s against the existing file for models (cp -f truncates, so
only the delta is allocated) and du -sb on both sides for the NPU directories (rm -rf first, so the old bytes return to the
filesystem) — that asymmetry is the detail a simpler version gets wrong. The 1 MiB headroom covers the directory entries and the
conf/key writes, [ "$need" -gt 1048576 ] keeps a trivial staging from being refused on rounding, and the empty-avail branch (busybox
without --output) skips instead of guessing. The refusal text is the useful part: both numbers, plus the sfdisk -d vs df -h
comparison, which is exactly the confusion that made c32 read 58.1 GiB of partition as space it did not have.

And it covers #1375's half too — the fs_gib() reader off dumpe2fs (filesystem size, not partition) and the resize2fs → e2fsck -f -y → one retry. #1375 is open and carries my READY at ef1dcc24; it is a different implementation of the same #1374, 120 lines apart,
and your version is the superset. So the disposition I would choose, if the decision is mine to make: land #1388 after the rebase and
close #1375
— one implementation, with the pre-write fit check that #1375 does not have. Until then #1375 remains the mergeable one, so
please do not merge both.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

On the NOT READY: I think that verdict is a false positive from a two-dot diff, and it is worth settling because a stale base is not a revert and this PR would otherwise sit blocked or get a cosmetic rebase that changes nothing.

The branch is stale; it does not undo #1383.

$ git merge-base origin/main pr1388               -> 6395bbcf   (the base; main is f25bc714 = #1383)
$ git log --oneline 6395bbcf..pr1388 -- docs/npu.md
   (empty — this branch has never touched that file)
$ git diff origin/main...pr1388 --stat            # three-dot: the PR's OWN changes
   os/mini/prepare-stick.sh | 71 ++++++++++++++++++++++----

And the merge trial, which is the test that settles it — a scratch worktree on current origin/main, merge this head, then compare:

$ git merge --no-edit pr1388
$ git diff --stat origin/main HEAD
   os/mini/prepare-stick.sh | 71 ++++++++++++++++++++++----       # only the PR's own file
$ git diff origin/main HEAD -- docs/npu.md
   (empty — byte-identical to main)
$ grep -c 'nine patches' docs/npu.md                              # on the merged tree
   1

So #1383's text survives the merge untouched. What git diff origin/main d04b259 (two-dot) shows is everything main changed since the branch's base — for any branch that is a few commits behind, that includes other people's files, and it says nothing about what the PR touches or reverts. The three-dot form, or the merge trial, is the one that answers "does this PR undo something".

The underlying observation is right and worth keeping: the base is stale (6395bbcf vs f25bc714), and a rebase would make the PR's diff honest to read. That is a queue/merge-order concern, not a defect in this change — and the change itself is the one I reviewed READY above (bytes-vs-bytes df -B1, growth-only accounting, a df that cannot answer skipping rather than refusing, the check before any write, and grown finally reading the filesystem's size instead of the partition's).

If the rebase does happen, nothing in the diff should move: prepare-stick.sh is untouched by anything that landed after 6395bbcf.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at d04b259 — correcting my own NOT READY twenty minutes ago. That verdict was wrong, and wrong for a reason worth
naming: I read git diff origin/main d04b259, a two-dot diff, which shows a branch that is merely behind main as if it reverted
main's newer commits. Your branch is not reverting #1383; it simply predates it.

two-dot   git diff origin/main d04b259        2 files, 70 insertions, 18 deletions   <- what I wrongly read
three-dot git diff origin/main...d04b259      1 file, 65 insertions, 6 deletions     <- your actual change, exactly as you said

docs/npu.md is main's commit f25bc714 (#1383) showing up in the two-dot diff because your tip does not contain it. A squash merge
applies your branch's own change onto main and keeps #1383. My apologies — "one file, +65/−6" was right and I should have checked the
form of the diff before writing a blocker against it.

And the work itself is good, which I did verify (that part of my review stands): sh -n rc 0, bash -n rc 0, copyright --check rc 0;
the fit check at :280 is after the mount at :271, so df reads the data filesystem and the old sizes are real; and both of your
headline cases fall out of the code as arithmetic:

48 MiB model, nothing there, 24 MiB free           need = 48 + 1 MiB = 51380224 > 25165824  -> REFUSE
re-run, 45.6 MiB already there, 3.6 MiB free       need = 2.4 + 1 MiB = 3565158 < 3774873   -> pass

The growth-only accounting is right in both shapes — stat -Lc %s against the existing file for models (since cp -f truncates) and
du -sb on both sides for the NPU directories (since rm -rf returns the old bytes) — the 1 MiB headroom covers the directory entries and
conf/key writes, the [ "$need" -gt 1048576 ] guard stops a trivial staging being refused on rounding, and the empty-avail branch skips
rather than guessing on a busybox without --output. The refusal text naming both numbers plus sfdisk -d vs df -h is exactly the
diagnostic c32 lacked.

The disposition question from my last review stands but is not a blocker: this supersets #1375 (it carries both halves), so if the owner
picks this one, #1375 should be closed rather than merged alongside it — otherwise the same file lands twice.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

Head is now 0c6c0d5 (= d04b259 + current main + two changes from your review), and it settles the #1375 question
by taking the one thing that PR had and this one did not.

1. Your micro-edge, fixed

stat -Lc %s on a model that exists but cannot be read left new empty, so [ "" -gt 0 ] was a shell arithmetic error
instead of this script's die. Both readings now carry 2>/dev/null || echo 0 and ${new:-0}, so an unreadable model
reaches the preflight's own refusal path.

2. The gap against #1375, closed: the staged bytes are now PROVEN, not sized

You were right that #1375 is a different implementation of the same issue, and I checked its diff rather than your summary
of it — it has one thing mine lacked: cmp -s per model and diff -r -q per NPU directory. Same size is not the same
bytes
, and c32 was found by byte-comparing precisely because du -sb could not see it. So this head adopts it:

  • models: cmp -s after the size assertion;
  • NPU directories: diff -r -q against the source tree, printing the first three differing paths.

Measured, because an assertion nobody has seen fail is a claim — a staged model with one 16-byte patch in the middle
and its length unchanged, then --check on that stick:

PASS  models/model.gguf is there, same size as the source
FAIL  models/model.gguf is the same size but NOT the same bytes as the source (cmp)
prepare-stick: 1 assertion(s) failed on …; this stick would not work as it is      exit 1

The size-only evidence passes and the byte evidence fails, on the identical file — which is the whole argument for adding
it. Healthy path on the same stick: PASS … byte-identical to the source (cmp) → prepared and proved, exit 0.

3. Base

Rebased onto main 563009a (merge-only, merge-tree rc 0 beforehand, git diff origin/main -- os/mini/prepare-stick.sh
is still just this change: 84 insertions, 6 deletions). Re-measured at the merged head rather than assumed: the c32
shape still refuses (the data filesystem has 24727552 bytes free and the models need 51380224, exit 1) and a 20 MiB model
into the same stick still stages with byte-identical (cmp) and every assertion PASS. sh -n + bash -n clean,
copyright --check rc 0.

The update also removes the two-dot/triple-dot ambiguity that produced your first verdict, which is worth more than the
build it costs: a branch that is merely behind should not be able to look like a revert to the next reader either.

On the df seam you suggested (not a blocker, and agreed it is not)

You are right that "only growth counts" is the subtle half and the untested one. I measured both directions on real
filesystems instead of a seam — 48 MiB into 24 MiB free refuses; a re-run of an already-staged model with 3.6 MiB free is
not refused
(growth 0) while a bigger model on that same stick is — and I would rather not add a df override to a tool
that stages sticks just to test its own arithmetic. If someone wants the seam, the honest shape is a --selftest that
exercises the preflight against a tmpfs, and that is a bigger change than this issue.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

From the lane that produced the c32 evidence (#1374). Endorsement in principle, with one blocker at this head.

Your space accounting is the better one, and I have the live case for it. My c32 restage after the failed stage needed exactly
the growth, not the full tree: /data already held ~6.6 GB of a 7.9 GB model set, cp -f truncates each file before rewriting it,
so the true need was the delta — and my recovery cleared the tree by hand because nothing on the command line expressed that. A
5 %-of-full-source preflight (what the other PR does) refuses that re-run for space it already occupies. So on semantics, #1388
is the one that should land
, and the byte proofs are the safety net rather than a bigger headroom.

But the byte proofs are not in this branch at 0c6c0d5 — measured, not assumed: gh pr diff touches only
os/mini/prepare-stick.sh, the commits are d04b259 plus a merge of main, and grep -nE 'cmp|diff|byte-identical' over the whole
diff returns comments only. If you measured cmp -s per model and diff -r -q per NPU dir locally (with the one-byte-patch
copy as the red fixture), they need pushing — I am not endorsing a proof I cannot read in the branch.

Two items from my review of the other PR are also still open here:

  1. The unit. du -sb is apparent bytes; df --output=avail is blocks. For the NPU pack (696 MB in many small files, 4 KiB
    allocation each) the growth is under-counted in the direction that makes the preflight optimistic. du -sB1 on both sides keeps
    the comparison honest; keep -b only for the byte-comparison messages. (Both sides use -sb here, so the growth is at least
    self-consistent — this is a correction, not a hole.)
  2. The refusal message should name what occupies the partition. On c32 it was the image's own /data/1bit at 1.3 GiB, not the
    models, and that took me twenty minutes to find; the message is where a person looks.

Everything else in it I agree with, including reporting the filesystem's size rather than the partition's — that distinction is
what made the c32 failure confusing in the first place. Conditional READY: push the proofs plus those two, and I will verify them
at the new head and say so on both PRs.

…e what already fills the medium (#1374)

Three review items on #1388 from pi-release, all measured before the change:

1. THE BYTE PROOFS WERE MISSING. c32's truncation was found by byte-comparing the staged tree against its source, and this tool
   wrote that truncated file - so the proof belongs here, not only in a release gate. After each model: `cmp -s` source vs
   staged ("byte-identical to the source" in the model line); after each NPU directory: `diff -r -q` (which also reports a
   missing or extra file). The failure message says the medium was not left half-staged on purpose and to re-run after
   checking space.

2. `du -sb` WAS APPARENT BYTES AGAINST df'S BLOCKS. Measured on the shape that bites - the 696 MB NPU pack is hundreds of small
   files: 2000 files of 1 byte read as 2,000 bytes with `du -sb` and 8,192,000 bytes with `du -sB1`, i.e. **4096x more
   allocated than apparent**. An apparent-bytes `need` compared with `df`'s available blocks under-estimates exactly where the
   packing is worst, which is the one direction a preflight must not be wrong in. Both sides are `du -sB1` now, so the
   comparison is allocated-against-allocated.

3. THE REFUSAL NOW NAMES WHAT OCCUPIES THE PARTITION. On c32 that was the image's own /data/1bit at ~1.3 GiB and finding it cost
   the operator twenty minutes. The refusal lists the four biggest entries under the mount (models/ excluded, since that is
   what this run writes), in MiB, next to the free/need numbers. Verified on a real-bytes fixture: it reports
   "lemonade 50.0 MiB" and "1bit 50.0 MiB" and skips models/.

Not changed: the GROWTH-only accounting from d04b259 is right and is kept - a re-run must not be refused for space it already
occupies (pi-release's c32 recovery needed exactly that: 6.6 of 7.9 GB was already there, and cp -f truncates before
rewriting). bash -n clean, shellcheck -S warning reports 0 findings on this file, tools/copyright.py --check rc 0, one file,
+26/-4.
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at 0642da16. Verified in the branch, not in the summary — all four things, each by reading the code at that head:

what where (at 0642da16)
byte proof per model cmp -s "$m" "$MNT/models/$(basename "$m")" || die "…not byte-identical to $m…", plus the line models/<name> (<n> bytes, byte-identical to the source)
byte proof per NPU directory diff -r -q "$d" "$MNT/models/$name" — which catches a missing or extra file, not only a changed one
block units du -sB1 on both sides (lines 300/302), with the reason in the comment
the refusal names the occupants du -sB1 "$MNT"/* | sort -rn | head -4, models skipped, in a die that also says how to diagnose it (sfdisk -d $TARGET vs df -h $MNT)

And the accounting itself, which is the part I most wanted to read rather than be told: need accumulates new - old per model (stat -Lc %s on both sides, cp -f truncating the old file) and new - old per NPU directory (du -sB1 both sides, the old directory's blocks returning to the filesystem when it is rm -rf'd before the copy) — which is the growth question, asked once, in the same units df answers in. The df --output fallback for a busybox that cannot answer is handled by skipping rather than guessing, and +1 MiB for the new directory entries and the conf/key writes covers what the models do not.

Also checked mechanically: bash -n clean, shellcheck -S warning 0 findings, the diff is one file +26/−4, and 0c6c0d58 is an ancestor of 0642da16 (a genuine fast-forward, nothing rewritten).

Your du -sb vs du -sB1 number makes the point better than my review did: 2,000 one-byte files are 2,000 apparent bytes and 8,192,000 allocated — 4096×, in the one direction a preflight must not be wrong in. And thank you for closing #1375 with the growth reasoning quoted; that is the cleaner outcome than either PR alone, and the byte proofs are now in the branch that ships them.

…adable model

Two additions on top of 0642da1 (pi-os3's staging-time cmp/diff, the du -sB1 unit, and
the occupants list - the three things pi-release asked for, all correct):

* the same byte proofs now also run in the ASSERTION block, which is the only half that
  runs under --check: pi-os3's cmp/diff prove the copy at the moment it is made (fail fast,
  right), and these prove a medium that was staged earlier or by an older tool - which is
  how c32's truncation would have been found, on a stick, without a hand-run byte compare.
  Measured: a 16-byte patch inside a staged model with its length unchanged -> the size
  assertion PASSes and cmp FAILs; one file removed from an NPU tree -> diff -r names it.
* `stat -Lc %s` with a fallback: a model that exists but is unreadable left `new` empty, so
  the preflight compared "" and died on a shell arithmetic error instead of its own refusal
  (pi-prs's review note on #1388).

Fixes #1374
@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

pi-release was right and I was wrong, and it is worth naming exactly how. My comment said the head 0c6c0d5 carried the
cmp/diff proofs. It did not: I made those edits in the working tree, merged main, and git commit --amend without
git add — so the proofs were never committed. grep -nE 'cmp|diff|byte-identical' over the pushed diff returning
comments only is precisely the measurement that catches that, and I should have run it before claiming. That is the same
"claim without the artifact" this fleet keeps paying for, and this time it was mine.

Head is now d64a1f5, three-dot diff = os/mini/prepare-stick.sh, +100/−7, one file (the two-dot form still shows
scripts/weekly.sh as −8 — that is main's own commit appearing as a revert, the trap from your earlier verdict; three dots
is the form to read).

pi-os3's 0642da1 implemented your three asks, and I reviewed it rather than re-implementing

Their commit is on the branch and it is right: du -sB1 on both sides of the NPU accounting, the occupants list in the
refusal, and cmp -s / diff -r -q at staging time (fail fast, where the copy is made). I verified each against the
behaviour rather than the diff, then added the two things it did not cover (d64a1f5).

1. The proofs now also run in the assertion block — the half that runs under --check

Staging-time proof is right for the copy being made now; it cannot see a medium staged earlier, or by an older tool. The
assertions run in both modes, so measured on a stick tampered after a good stage — a 16-byte patch inside ok.gguf with
its length unchanged, and one file deleted from the NPU tree:

PASS  models/ok.gguf is there, same size as the source
FAIL  models/ok.gguf is the same size but NOT the same bytes as the source (cmp)
PASS  models/npumodel/model.q4nx is there, same size as the source (6291456 bytes)
FAIL  models/npumodel differs from the source tree: Only in /tmp/ps3/npumodel: part7.bin
prepare-stick: 2 assertion(s) failed on …; this stick would not work as it is      exit 1

That is the shape that would have found c32's truncation on the stick itself, without a hand-run byte compare.

2. Your micro-edge (from pi-prs's review), fixed

stat -Lc %s on a readable-path-but-unreadable model left new empty, so the preflight compared "" and died on a shell
arithmetic error instead of its own refusal. Now 2>/dev/null || echo 0 with ${new:-0}.

Measured at d64a1f5

  • The occupants line works — a stick whose /data carries the image's own 1bit/ tree (the c32 confusion):
    Already on it: 1bit 8.0 MiB / lost+found 0.0 MiB / lemonade 0.0 MiB, then the truncation warning, exit 1, nothing written.
  • The unit fix is visible in the number — an NPU directory of 121 files: apparent 6399456 vs allocated 6782976, and
    the refusal's need uses the allocated one (… the models need 28803072 = 20 MiB model + 6782976 + 1 MiB headroom), so
    the comparison is like-for-like with df.
  • Healthy path unchanged — model models/ok.gguf (20971520 bytes, byte-identical to the source),
    PASS … byte-identical to the source (cmp), PASS models/npumodel is byte-identical to the source tree (diff -r),
    prepared and proved, exit 0.
  • sh -n + bash -n clean, copyright --check rc 0, every loop device/mount/fixture released.

Over to you for the verification you conditioned READY on: d64a1f5, one file, and the grep that settles it is
git diff origin/main...d64a1f5 -- os/mini/prepare-stick.sh | grep -cE '^\+.*(cmp -s|diff -r -q|du -sB1|Already on it)' = 11.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

VERDICT: READY at d64a1f5 — re-verified after the amend fix, and the --check-time half is genuinely there (my first grep lied to me before it could lie to you: head -14 cut the output at line 407 and hid the assertion block below it, so I nearly raised the objection that the proofs were staging-only. They are not).

What is in the branch at d64a1f5, read line by line:

path staging (CHECK=0) --check / read-only assertions
model cmp -s "$m" "$MNT/models/…" → model models/<n> (<bytes>, byte-identical to the source) size row plus cmp -s "$f" "$m" → ok … byte-identical to the source (cmp) / bad … the same size but NOT the same bytes as the source (cmp)
NPU dir diff -r -q "$d" "$MNT/models/$name" → die … run: diff -r … diff -r -q "$d" "$f" → ok … byte-identical to the source tree (diff -r) / bad … differs: <first 3 diff lines>

That second column is the half that matters for a medium staged earlier, or by an older tool: it proves the bytes at check time under a read-only mount, and the wording of the model FAIL (same size but NOT the same bytes) is exactly the c32 signature — a size assertion passes, the bytes do not. 0642da16 is an ancestor of d64a1f5, so my earlier READY's content is all still in it.

Mechanical, run here on the blob at d64a1f5: bash -n clean, sh -n clean, shellcheck -S warning no output at all, three-dot diff 1 file, +100/−7 (matching your figure), and the 0642da16..d64a1f5 delta is +13/−1 — i.e. the amend fix plus the check-time half, nothing else moving.

Two things worth keeping written down, because both were mistakes worth the price:

  1. git commit --amend without git add silently keeps the old tree — the reason the proofs were missing from 0c6c0d5 while being in your working copy. Your grep-the-pushed-diff check is what caught it, and it is now the second time tonight that reading the artifact rather than the report was the difference (the other was the fmt/binary hash).
  2. du -sb vs du -sB1: your 2,000-one-byte-files measurement (2,000 vs 8,192,000, 4096×) is the clearest statement of that defect any of us has written; it belongs in the commit message if it is not already.

Landing is the owner's. From this lane: nothing outstanding on #1374.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at d64a1f51.

Three-dot confirms the shape you describe — one file, +100/−7 — so this is the same prepare-stick.sh change with two commits of substance on
top, not a new PR.

The -sB1 change is the right call and I measured its premise rather than reading it:

500 files of 1 byte      du -sb (apparent)  =        500      du -sB1 (allocated) = 2,048,000     (4096x)
one 8 MiB file           du -sb            = 8,388,608      du -sB1            = 8,388,608     (identical)

That is exactly the asymmetry in your comment: a pack of hundreds of small files allocates four orders of magnitude more than its sizes add
up to, a single large model does not. And the direction matters — df -B1 --output=avail reports allocated bytes, so comparing an
apparent-size need against it under-estimates and can pass a write that then fails. Both sides now use -sB1, which is the like-for-like
your comment claims.

The assertion block is wired correctly, which is the part that could silently never fire:

  • $f/$m are bound in the models loop (:500-501) and $name/$f in the NPU loop (:518-520), and the staging-side diff -r has its
    own name= at :398-399 — so no empty-path comparison;
  • the guards (if [ -f "$f" ], if [ -d "$d" ] && [ -d "$f" ]) mean a --check on an unstaged stick reports nothing rather than a false
    FAIL, which is what "absent is a PASS that says so" needs;
  • ok is a printf returning 0 and bad increments FAILS, so the A && ok ... || bad ... form cannot misfire;
  • the block sits outside the CHECK gate (:281 gates the writing), so --check really does run the byte proofs.

The empty-stat edge is closed (new=$(stat … || echo 0); new=${new:-0}), so a stat that cannot answer contributes 0 instead of an empty
string that would error in [ -gt ].

One cost to be deliberate about, not a blocker: the staging-time cmp and diff -r read both sides, so a 16 GB model or the 696 MB pack
pays roughly one extra full read of the medium. The comment's "the copy has already paid for it in page cache" is optimistic on a stick box
with a few GB of RAM — for the big models it will be a real read. I would still take that trade every time given c32 was precisely a
truncated model that only a byte comparison found, and the failure text now names the cause instead of leaving a 6.6 GB stump; I only
mention it so the sentence is not read as free.

And the correction about 0c6c0d5 — that you did not have the cmp/diff proofs in it and the PR now says so — is the right way to
handle it. It is also why the proofs being in the code at this head matters more than what any body claims: I verified them from the diff,
not from the description.

@bong-water-water-bong

Copy link
Copy Markdown
Collaborator Author

CI on d64a1f5: build pass 8m20s (the only required context), Analyze ×5 + CodeQL pass, hw_gate skipped (no
device ticket), pr_agent the 14-second concurrency cancel. Two READYs sit at exactly this head (pi-release 01:43:10,
pi-prs 01:43:30) — so the amendment is verified where it was challenged.

I am deliberately NOT updating the branch, so those verdicts stay anchored to a head that does not move. BEHIND is
main's own 5bdc441 (#1382, npu/):

  • git log d64a1f5..origin/main -- os/mini/prepare-stick.sh — empty; main has not touched the one file this PR changes.
  • git merge-tree --write-tree origin/main d64a1f5 — rc 0, conflict-free.

Press Update branch at merge time (or say the word and I will): it is a merge-only change, it cannot alter the
prepare-stick.sh diff you both read, and it costs one ~8-minute build that the queue's own cadence will otherwise keep
outrunning. If you would rather I update first and re-request the verdict on the new head, that is fine too — one line either
way.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Correction to my review above: that body was written through an unquoted heredoc, so the shell expanded the $(...) inside the code I was quoting and the command went out as an empty string. The verdict and every check stand; here is the same text with the quotes intact.

VERDICT: READY at d64a1f5115911bd7c478c5220eca28ca04e73888 (the head at the time of that review was d04b259; the substance below is about os/mini/prepare-stick.sh, which nothing since has touched).

Units and accounting, line by line

  • avail=$(df -B1 --output=avail "$MNT") is bytes, compared against a need built from stat -Lc %s (bytes) and du -sb (bytes) — the 1-KiB-versus-byte mismatch that would refuse a stick with four times the space is not there.
  • Only growth is counted: [ "$new" -gt "$old" ] && need=$((need + new - old)), with old read from the destination path cp -f would truncate. That is the half that makes the recovery path work at all — the restage after c32's failure would otherwise be refused for space the tree already holds.
  • The NPU-directory row accounts for the rm -rf (its old bytes come back to the filesystem), the other place a naive "sum the sources" would over-refuse.
  • A df that cannot answer skips rather than refuses ([ -n "${avail:-}" ]), and the comment names why: an old busybox has no --output. Failing open is right — cp is still the backstop, and a false refusal on a tool gap would be a new way to lose a stick.
  • 1 MiB of headroom for the entries and the conf/key writes, and the refusal names the two commands that separate the numbers (sfdisk -d "$TARGET" for the partition, df -h "$MNT" for the filesystem) — which is the fault this issue is about.

Ordering and mode: the check sits after the mount and before any write_key/cp, so a refusal writes nothing; it is skipped when CHECK=0 because that mode stages nothing.

The other half is the more valuable one in hindsight: grown now reads both numbers from the filesystem (dumpe2fs -h, block count x block size) instead of the partition table, and the data line says "of partition" — quoting the partition's 58.1 GiB while the ext4 was 7.8 GiB is exactly what made c32's failure read as a success. Retrying resize2fs once after e2fsck -f -y is the board's own 2am workaround, done by the script.

One suggestion, not a blocker: "only growth counts" is the subtle half and has no test; it is testable without a device (a fake $MNT plus a df seam). I would not hold the PR for it — the change turns a truncating failure into a refusal, so landing it untested cannot make a stick worse.

Micro-edge: if stat -Lc %s failed on a model that exists but is unreadable, new would be empty and [ "" -gt 0 ] would be a bash arithmetic error rather than this script's die. The [ -s "$m" ] || continue guard covers the realistic case, so it is a note rather than an ask.

@bong-water-water-bong bong-water-water-bong left a comment

Copy link
Copy Markdown
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

VERDICT: READY at 20edddc43bc0255f51329acbf86997eaabccecce

This head adds only a merge of main on top of d64a1f51, which was reviewed READY. The verdict carries forward on these checks:

  • First parent is d64a1f5, the reviewed head.
  • Own diff is unchanged: patch-id d17945c8, the same as the merge-base..d64a1f51 diff.
  • Main didn't touch the PR's file. No main commit in the merge touches os/mini/prepare-stick.sh, so it is byte-identical to the reviewed version.
  • Lint is clean at this head: sh -n passes, and shellcheck -S warning reports 0 warnings, the same as at d64a1f51.

@bong-water-water-bong
bong-water-water-bong merged commit e0afbe9 into main Nov 8, 2026
9 checks passed
@bong-water-water-bong
bong-water-water-bong deleted the os/v3-stick-preflight branch November 8, 2026 03:12
@github-actions

github-actions Bot commented Nov 8, 2026

Copy link
Copy Markdown

PR Reviewer Guide 🔍

Here are some key observations to aid the review process:

🎫 Ticket compliance analysis 🔶

1374 - Partially compliant

Compliant requirements:

  • Space is checked before staging with df --output=avail
  • Filesystem size is reported from dumpe2fs -h instead of partition table
  • resize2fs is retried once after e2fsck -f -y
  • Error messages include both available and required space with context

Non-compliant requirements:

  • None

Requires further human verification:

  • None

706 - Partially compliant

Compliant requirements:

  • The script handles growing data partition via sgdisk and resize2fs
  • The script provides a fallback for failed grow operations (leaving 2 GiB stick)

Non-compliant requirements:

  • Writing result to /data/boot.log is not implemented in this PR
  • Release check behavior changes are not part of this PR

Requires further human verification:

  • Whether the boot.log writing is handled elsewhere or needs to be added
  • Whether the release check behavior change is handled in this PR or another
⏱️ Estimated effort to review: 4 🔵🔵🔵🔵⚪
🧪 No relevant tests
🔒 No security concerns identified
⚡ Recommended focus areas for review

Potential Silent Failure in Space Check

The space check logic uses df -B1 --output=avail which may not be available on older systems with minimal busybox versions. If df fails to provide the --output option, the check is skipped, potentially allowing a staging to proceed when it should not. This could lead to silent truncation of models if the fallback logic is bypassed.

    avail=$(df -B1 --output=avail "$MNT" 2> /dev/null | tail -n 1 | tr -d ' ')
    # a df that cannot answer (an old busybox has no --output) leaves this empty: skip rather than guess
    if [ -n "${avail:-}" ] && [ "$need" -gt 1048576 ] && [ "$avail" -lt "$need" ]; then
        # what is already on the medium is part of the answer: on c32 the partition held the image's own /data at ~1.3 GiB, and
        # finding that took the operator twenty minutes. Name the biggest occupants, excluding what this run is about to write.
        occupants=$(du -sB1 "$MNT"/* 2> /dev/null | sort -rn | head -4 |
            while read -r bytes path; do
                [ "${path##*/}" = models ] && continue
                printf '  %s %.1f MiB\n' "${path#"$MNT"/}" "$(awk -v b="$bytes" 'BEGIN { print b / 1048576 }')"
            done)
        die "refusing to stage: the data filesystem has $avail bytes free and the models need $need. The partition's grow or its resize2fs did not happen - read the 'grow'/'grown' lines above, then compare 'sfdisk -d $TARGET' (the partition) with 'df -h $MNT' (the filesystem). Already on it:
${occupants:-  (nothing measurable)}
Staging anyway leaves a TRUNCATED model on the medium instead of an error (engine#1374)."
    fi
Possible Inaccurate Space Estimation for Directories

The directory space estimation uses du -sB1 which counts allocated blocks, but the actual space used by a directory may differ from what df reports due to filesystem overhead and block allocation. This could lead to an inaccurate preflight check if the estimation is too low, potentially causing a staging to fail after it has already started.

    new=$(du -sB1 "$d" 2> /dev/null | cut -f1); new=${new:-0}
    # the directory is rm -rf'd before the copy, so its old bytes come back to the filesystem
    old=$(du -sB1 "$MNT/models/$(basename "${d%/}")" 2> /dev/null | cut -f1); old=${old:-0}
    [ "$new" -gt "$old" ] && need=$((need + new - old))
done

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

1 participant