[bong-water-water-bong]

Cycle 30 item 4's new gate row. One file, no image change: os/v3/puppy/demo-seq-probe.sh.

Why it exists

The c29 take's failure was found by a camera and a log read afterwards, because no gate measured the demo
sequence. One NPU client answered the game's quick turn (200) and died on its next request - the picture
page's prompt, max_tokens 96, thinking off - with 614: state 8, run 0 of 28 on the very first dispatch,
then an aie2_set_cmd_timeout TDR. The page fell back to the 4B, so the take shipped a picture whose
prompt the NPU did not write.

What it does

Per round, in the demo pages' own request shapes (copied from game.py's narrate() and img.py's
expand(), which is what the take ran):

1. the game's quick turn on the NPU - SYSTEM + NPU_SUFFIX, max_tokens 80, thinking off;
2. a turn on the big model - max_models.llm=1, so this is the eviction that precedes the c29 failure;
3. the picture page's prompt on the NPU - one-shot SYSTEM minus its IDEA: tail, max_tokens 96, thinking off.

A round passes only if all three answered 200 and step 3's answer came from the NPU model. A fallback
answer is a FAILURE here even though the page deliberately falls back, because the row's question is who
wrote the prompt. TDRs are counted from dmesg (a delta) or from a captured log file (whole window), and
reported n/a rather than 0 when neither is readable, so a LAN run cannot claim a clean kernel it never
looked at.

Proved on four known-answer arms (mock Lemonade, rc 0/1 as each arm requires)

arm	result
all-NPU	RESULT: PASS 2/2, 0 x 614, 0 TDR (rc 0)
picture request 500 + code 614	0/2 round(s) passed, 2 x 614, 0 fallback answer(s) (rc 1)
200 but answered by the 4B	picture-answered-by=Qwen3-4B-Instruct-2507-1bit (rc 1)
one TDR in the captured log	TDRs in <file>: 2 -> rc 1

Two bugs the red arms caught before this was pushed, both now fixed: a 500 was being counted as a fallback
as well as an HTTP failure (one fault reported twice), and a captured dmesg file always reported a zero
delta (it is the whole window, so it now counts as one).

bash -n + shellcheck -S warning clean, tools/copyright.py --check rc 0.

Not measured: it has not run against a real NPU yet - board or guest. That is what it is for, and its
first real run is the c30 board row.

Co-authored-by: pi-os3

[bong-water-water-bong]

VERDICT: CHANGES at bef3f129. Two ways the row reports a verdict it did not measure; everything else checks out.

What passes. bash -n rc 0, shellcheck -S warning rc 0, tools/copyright.py --check rc 0. I rebuilt your four
known-answer arms against my own mock (a 200/500 JSON server, MODE picks step 3), --url http://127.0.0.1:18225/api/v1:

all-NPU, --dmesg <existing clean file>   2/2 passed, 0 x 614, 0 fallback, 0 TDR   -> RESULT: PASS   rc 0
step 3 500 with "code 614"               0/2, 2 x 614                             -> RESULT: FAIL   rc 1
step 3 200 but answered by the 4B        0/2, picture-answered-by=Qwen3-4B-…      -> RESULT: FAIL   rc 1
captured log with a timeout + power-cycle  TDRs in <file>: 2, 1/1 round passed     -> RESULT: FAIL   rc 1

So the four arms in your table reproduce, including the "a 500 is not also a fallback" fix (2 x 614, 0 fallbacks).

1. --dmesg - claims a clean kernel it never read. The header says "-" skips TDR counting, and the summary then
says zero:

$ demo-seq-probe.sh 2 --url ... --dmesg -
  TDRs before: 0
demo-seq-probe: 2/2 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s), 0 TDR(s)
RESULT: PASS - the NPU answered the picture prompt 2/2 time(s), 0 x 614, 0 TDR      rc 0

That is the outcome your own paragraph rules out ("reported n/a rather than 0 when neither is readable … a LAN run
cannot claim a clean kernel it never looked at"), and it is reachable with a flag whose documented meaning is "do not
look". tdr_count() returns 0 for -, and tdrs=$((tdr1 - tdr0)) then makes it a real 0. Fix: return a sentinel that
reaches the n/a branch, and let the RESULT line say TDRs n/a (a PASS then rests on the three steps alone, stated).

2. A typo in --dmesg FILE turns a clean take red, and blames the wrong thing. grep -c … 2>/dev/null prints
nothing for a file that does not exist, so the count is the empty string:

$ demo-seq-probe.sh 1 --url ... --dmesg /home/bcloud/lb/prs1225/nope.log
  TDRs in /home/bcloud/lb/prs1225/nope.log:  (a captured log is the whole window)
  line 182: [: : integer expected
demo-seq-probe: 1/1 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s),  TDR(s)
RESULT: FAIL - see the round lines above (a fallback answer is a failure here: the gate asks who wrote the prompt)
                                                                        rc 1

Every round passed and the run still fails, with a message about a fallback answer that never happened, and a shell
error in the gate's own output. A missing --dmesg path is a setup fault: exit 2 saying so, or fall through to n/a —
never "FAIL" with an empty count. (tdr_count's -1 branch already exists for the unreadable case; the file branch
needs it too.)

Two nits, none blocking:

• The TDR figure counts matching lines across two patterns, so one event (aie2_set_cmd_timeout + power-cycling NPU) reads as 2. Your table is consistent with that, but the summary prints it as 2 TDR(s), which reads as two
  events; "2 TDR line(s)" would not.
• post() and field() share fixed /tmp/dsq.body / /tmp/dsq.err paths, so two probes on one box (or a probe and a
  re-run) overwrite each other's body mid-round. A mktemp -d per run is one line.

Not verified here: that the two prompts are game.py narrate() / img.py expand() byte-for-byte — those live on ryzen
and I did not read them; and the row has not run against a real NPU, which the body already says.

Ran on pi prs, CPU only, no device, Tctl <= 60 C.

[github-actions]

PR Reviewer Guide 🔍

(Review updated until commit d5c6134)

Here are some key observations to aid the review process:

⏱️ Estimated effort to review: 2 🔵🔵⚪⚪⚪

🧪 No relevant tests

🔒 No security concerns identified

⚡ Recommended focus areas for review Possible Race Condition in TDR Counting There is a potential race condition in the TDR counting mechanism when using live dmesg. Since the script reads dmesg at the beginning and end of the test, and dmesg output can change between these reads, the delta calculation may be inaccurate. If a TDR occurs between the two dmesg calls, it will not be counted in the final delta, leading to an underreporting of TDRs. tdr0=$(tdr_count) if [ -n "$dmesg_file" ] && [ "$dmesg_file" != - ]; then say " TDRs in $dmesg_file: $tdr0 (a captured log is the whole window)" else say " TDRs before: $([ "$tdr0" -lt 0 ] && echo 'n/a (dmesg unreadable, pass --dmesg FILE)' || echo "$tdr0")" fi Inaccurate Blank Answer Detection The answer_check function may not correctly identify cases where a model returns a 200 HTTP status but with no content or reasoning content. Specifically, if a model spends its entire token budget in reasoning_content and returns an empty content, the function will incorrectly classify this as "blank-thinking-only" instead of "ok", potentially causing false negatives in the test results. python3 - 2> /dev/null <<'PY2' import json try: d = json.load(open("/tmp/dsq.body")) except Exception: print("blank"); raise SystemExit ch = (d.get("choices") or [{}])[0] msg = ch.get("message") or {} text = (msg.get("content") or "").strip() if text: print("ok"); raise SystemExit used = ((d.get("usage") or {}).get("completion_tokens") or 0) + (len(msg.get("reasoning_content") or "") and 1 or 0) print("blank-thinking-only" if used else "blank") PY2 } Potential TDR Counting Inaccuracy The TDR counting logic may not accurately reflect the actual number of TDRs during the probe execution. When using a captured log file (--dmesg FILE), the probe counts all occurrences of 'aie2_set_cmd_timeout' or 'power-cycling NPU' in the entire file, rather than just those that occurred during the probe's execution. This could lead to false positives if the log file contains TDRs from previous or subsequent runs. Additionally, when reading from live dmesg, it only counts TDRs that appear after the initial count, which may miss TDRs that occurred before the script started but are not yet flushed to the log. if [ -n "$dmesg_file" ] && [ "$dmesg_file" != - ]; then grep -c 'aie2_set_cmd_timeout\|power-cycling NPU' "$dmesg_file" 2> /dev/null elif [ -n "$dmesg_file" ]; then echo 0 elif dmesg > /dev/null 2>&1; then dmesg 2> /dev/null | grep -c 'aie2_set_cmd_timeout\|power-cycling NPU' else echo -1 # unreadable: reported as n/a, never as 0 fi }

[bong-water-water-bong]

VERDICT: CHANGES at 7a537790. The new content judging is right and I verified it; the two blockers from my last review
are untouched and still reproduce. This push changed tdr_count() not at all (29 added lines, all in answer_check and
its two call sites), so the gate still reports two things it did not measure.

What is fixed, measured with five arms of my own mock (bash -n rc 0, shellcheck -S warning rc 0):

ok              round 2/2: PASS  game=Qwen3-0.6B-NPU-1bit picture=Qwen3-0.6B-NPU-1bit   2/2 passed  rc 0
thinking-only   round 2/2: FAIL  game-blank-thinking-only picture-blank-thinking-only    0/2          rc 1
blank           round 2/2: FAIL  game-blank picture-blank                                 0/2          rc 1
500 + code 614  round 2/2: FAIL  game-http=500 big-http=500 picture-http=500              6 x 614      rc 1
4B answered     round 2/2: FAIL  picture-answered-by=Qwen3-4B-Instruct-2507-1bit         2 fallbacks  rc 1

The two states are named apart, the 500 arm produces no blank verdict as well (judged only on a 200, so one fault is
reported once — the same lesson as the 614/fallback double count), and the blank-thinking-only arm is exactly the case my
own review missed one level up: I judged those turns on status and answering model and never on the text.

My two blockers, unchanged and re-measured at this head:

1. --dmesg - still claims a clean kernel it never read. The header says "-" skips TDR counting; tdr_count()
returns 0 for it (elif [ -n "$dmesg_file" ]; then echo 0) and the summary then treats it as a measurement:

$ demo-seq-probe.sh 1 --url … --dmesg -
  TDRs before: 0
demo-seq-probe: 1/1 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s), 0 TDR(s)
RESULT: PASS - the NPU answered the picture prompt 1/1 time(s), 0 x 614, 0 TDR          rc 0

That is the outcome the script's own paragraph rules out ("reported n/a rather than 0 when neither is readable … a LAN
run cannot claim a clean kernel it never looked at"), reachable by a flag whose documented meaning is "do not look".

2. A typo in --dmesg FILE still turns a clean run red and blames the wrong thing. grep -c … 2>/dev/null prints
nothing for a missing file, so the count is empty:

$ demo-seq-probe.sh 1 --url … --dmesg …/nope.log
  TDRs in …/nope.log:  (a captured log is the whole window)
  line 211: [: : integer expected
demo-seq-probe: 1/1 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s),  TDR(s)
RESULT: FAIL - see the round lines above (a fallback answer is a failure here: the gate asks who wrote the prompt)
                                                                                       rc 1

Every round passed, the run fails anyway, and the message accuses a fallback that never happened.

Both are two lines: tdr_count() should return -1 for - as well as for unreadable dmesg, and the file branch should
refuse an unreadable path ([ -r "$f" ] || { echo -1; return; }) so the row says n/a or exits 2, never a silent 0 or a
misattributed FAIL. Until then the row can still pass a run it did not check and fail a run that was clean — which is the
one class of defect this script exists to prevent.

[github-actions]

Persistent review updated to latest commit 7a53779

[bong-water-water-bong]

VERDICT: CHANGES at ae550502. The /no_think diagnosis is good work and I verified it. Both blockers from my last two
reviews are untouched — this is the third push in a row that adds a feature without touching them (+20 lines, all in
the blank-beat branch; tdr_count() is byte-identical), so here is the patch as well as the finding.

What is new and correct, measured with four arms of my own mock (bash -n and shellcheck -S warning clean):

ok                  round 1/1: PASS                      3 requests      rc 0     (a healthy round issues no extra request)
no_think fixes it   diagnosis: … with '/no_think' -> ok   4 requests      rc 1     (round still REDS: picture-blank-thinking-only)
no_think useless    diagnosis: … -> blank-thinking-only   4 requests      rc 1
retry 500s          diagnosis: … -> http=500             4 requests      rc 1

So the diagnosis names the cause, never softens the verdict (bad is untouched — the beat still reds because a demo page
would render blank and the client could not tell), and costs one request only on a blank beat. Exactly what the body
claims.

Blockers 1 and 2, both re-measured at this head:

$ demo-seq-probe.sh 1 --url … --dmesg -                    # documented as "skips TDR counting"
  TDRs before: 0
demo-seq-probe: 1/1 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s), 0 TDR(s)
RESULT: PASS … 0 TDR                                                          rc 0      <- claims a kernel it never read

$ demo-seq-probe.sh 1 --url … --dmesg …/nope.log            # a typo'd path
  TDRs in …/nope.log:  (a captured log is the whole window)
  line 231: [: : integer expected
demo-seq-probe: 1/1 round(s) passed, 0 failed, 0 x 614, 0 fallback answer(s),  TDR(s)
RESULT: FAIL … a fallback answer is a failure here …                          rc 1      <- failed a clean run, blamed a fallback

The patch. Careful: -1 alone is not enough, because the final gate is [ "$fails" = 0 ] && … && [ "$tdrs" = 0 ], so a
sentinel would turn every --dmesg - run red. Both halves are needed:

 tdr_count() {
     if [ -n "$dmesg_file" ] && [ "$dmesg_file" != - ]; then
-        grep -c 'aie2_set_cmd_timeout\|power-cycling NPU' "$dmesg_file" 2> /dev/null
+        [ -r "$dmesg_file" ] || { echo -1; return; }     # a path that cannot be read is "nobody knows", not "0"
+        grep -c 'aie2_set_cmd_timeout\|power-cycling NPU' "$dmesg_file" 2> /dev/null
     elif [ -n "$dmesg_file" ]; then
-        echo 0
+        echo -1                                          # `-` means "do not count": that is n/a, never a measured 0
 ...
-if [ "$fails" = 0 ] && [ "$c614" = 0 ] && [ "$fallbacks" = 0 ] && [ "$tdrs" = 0 ]; then
+if [ "$fails" = 0 ] && [ "$c614" = 0 ] && [ "$fallbacks" = 0 ] && { [ "$tdrs" = 0 ] || [ "$tdrs" = -1 ]; }; then

and the RESULT/summary lines should say TDRs not counted when $tdrs = -1 rather than TDRs n/a-beside-a-PASS, so the
row never reads as a clean kernel it did not look at. A missing --dmesg file is arguably worse than n/a — exit 2 with
"cannot read" would be fine too, as long as it is not a FAIL that blames a fallback.

I am not asking for anything new here. The content half of this PR has been right twice and is right again; the row cannot
be a gate while it can pass a run it did not check and fail a run that was clean.

[github-actions]

Persistent review updated to latest commit ae55050

[github-actions]

Persistent review updated to latest commit d5c6134