feat(cli): say when a generation went entirely to the reasoning channel

docs/benchmarks.md

@@ -455,6 +455,51 @@ the needle went unfound. Length selected which frequency band was live exactly
 the way forward width selects which kernel is dispatched above, and the same
 mistake is available in both.
 
+### The output half of an A/B
+
+A throughput arm says the change is faster. It does not say the model still
+answers the same, and that half has three ways of reading as a result when it
+is nothing of the kind. `scripts/ab_output_equality.sh` runs it so none of them
+is available:
+
+```bash
+git stash && cargo build --release --features metal,accelerate --bin mlxcel
+/bin/cp target/release/mlxcel target/release/mlxcel.before
+git stash pop && cargo build --release --features metal,accelerate --bin mlxcel
+./scripts/ab_output_equality.sh --baseline target/release/mlxcel.before \
+                                --arm target/release/mlxcel \
+                                --model models/mlx/granite-4.0-h-tiny-4bit
+```
+
+**Sampling makes the comparison meaningless in both directions.** A checkpoint's
+`generation_config.json` can turn sampling on with no flag from the caller, and
+then the two files being compared are two samples rather than two
+implementations: an untouched arm reads as a failure, and a genuinely broken one
+can pass. The script passes `--temp 0` to both arms and never takes it from the
+caller.
+
+**A blank content channel is not a blank generation.** `mlxcel generate`
+suppresses the `<think>` channel by default, so a reasoning model whose
+generation ends before the channel closes prints nothing, and comparing empty
+against empty passes while comparing no tokens at all. Read the other way it is
+worse: the blank looks like a broken checkpoint, or like breakage caused by the
+arm under test. That reading was one step away twice in one day, on
+`glm-4.1v-9b-thinking-4bit` and then on
+`nvidia-nemotron-3-nano-30b-a3b-4bit` during the RMS-norm A/B, where it would
+have inverted the verdict. The script passes `--show-reasoning` to both arms, so
+every generated token is in the comparison. The CLI now names the case as well:
+when tokens were generated and none reached the content channel, `generate` and
+the chat REPL print `[All N generated tokens went to the reasoning channel ...]`
+instead of an empty line.
+
+**An output difference is only attributable to the arm if the baseline agrees
+with itself.** Some families are not bitwise stable run to run (the f16
+reduction-order jitter class), and on those a difference between arms says
+nothing about the change. The script runs the baseline twice as a control and
+reports `INCONCLUSIVE`, exit status 2, when the two baseline runs disagree,
+rather than reporting the arm as different. On such a checkpoint the
+teacher-forced logit trace above is the tool, not this one.
+
 ### Gemma 4 Unified (12B) + 4-bit assistant
 
 `mlx-community/gemma-4-12b-it-4bit` as the target and

scripts/ab_output_equality.sh

@@ -0,0 +1,180 @@
+#!/usr/bin/env bash
+# Output-equality gate for an A/B arm.
+#
+# The throughput half of an A/B says whether a change is faster. This is the
+# other half: whether it still produces the same text. It exists because doing
+# that comparison by hand keeps reintroducing the same two errors.
+#
+#   1. Sampling. `generation_config.json` can enable sampling on its own, and
+#      then the comparison is between two samples, not two implementations, so
+#      an unchanged arm "fails" and a broken one can pass. Both arms run at
+#      `--temp 0` here, always.
+#   2. The reasoning channel. `mlxcel generate` suppresses `<think>` content by
+#      default, so a reasoning model whose generation is cut off inside the
+#      channel prints an empty content channel. Comparing empty against empty
+#      passes while comparing nothing at all, and reading it as breakage has
+#      twice cost an afternoon (glm-4.1v, then nvidia-nemotron-3-nano-30b-a3b
+#      during the RMS-norm A/B). Both arms run with `--show-reasoning` here, so
+#      the comparison covers every generated token.
+#
+# A third error is structural rather than per-run: an output difference is only
+# attributable to the arm if the baseline agrees with ITSELF. Some families are
+# not bitwise stable run to run (the f16 reduction-order jitter class), and on
+# those a difference between arms means nothing. The script therefore runs the
+# baseline twice as a control and reports INCONCLUSIVE, not FAIL, when the
+# control disagrees.
+#
+# Usage:
+#   ./scripts/ab_output_equality.sh --baseline target/release/mlxcel.before \
+#                                   --arm target/release/mlxcel \
+#                                   --model models/mlx/granite-4.0-h-tiny-4bit
+#
+#   --model / --prompt repeat, and every (model, prompt) pair is checked.
+#
+# Producing the baseline binary: build at the unpatched commit, copy the binary
+# aside, apply the patch, rebuild.
+#
+#   git stash && cargo build --release --features metal,accelerate --bin mlxcel
+#   /bin/cp target/release/mlxcel target/release/mlxcel.before
+#   git stash pop && cargo build --release --features metal,accelerate --bin mlxcel
+#
+# Exit status: 0 all pairs equal, 1 at least one pair differs, 2 at least one
+# pair inconclusive (unstable baseline) and none differ.
+
+set -uo pipefail
+
+BASELINE_BIN=""
+ARM_BIN="./target/release/mlxcel"
+MAX_TOKENS=128
+OUT_DIR=""
+MODELS=()
+PROMPTS=()
+
+usage() {
+  sed -n '2,45p' "$0" | sed 's/^# \{0,1\}//'
+  exit "${1:-0}"
+}
+
+while [[ $# -gt 0 ]]; do
+  case "$1" in
+    --baseline) BASELINE_BIN="$2"; shift 2 ;;
+    --arm) ARM_BIN="$2"; shift 2 ;;
+    --model) MODELS+=("$2"); shift 2 ;;
+    --prompt) PROMPTS+=("$2"); shift 2 ;;
+    -n|--max-tokens) MAX_TOKENS="$2"; shift 2 ;;
+    --out-dir) OUT_DIR="$2"; shift 2 ;;
+    -h|--help) usage 0 ;;
+    *) echo "unknown argument: $1" >&2; usage 1 ;;
+  esac
+done
+
+if [[ -z "$BASELINE_BIN" ]]; then
+  echo "error: --baseline is required (the binary built without the change)" >&2
+  exit 1
+fi
+for bin in "$BASELINE_BIN" "$ARM_BIN"; do
+  [[ -x "$bin" ]] || { echo "error: not an executable: $bin" >&2; exit 1; }
+done
+if [[ "$(cd "$(dirname "$BASELINE_BIN")" && pwd)/$(basename "$BASELINE_BIN")" \
+      == "$(cd "$(dirname "$ARM_BIN")" && pwd)/$(basename "$ARM_BIN")" ]]; then
+  echo "error: --baseline and --arm are the same file; the comparison would be vacuous" >&2
+  exit 1
+fi
+if [[ ${#MODELS[@]} -eq 0 ]]; then
+  echo "error: at least one --model is required" >&2
+  exit 1
+fi
+if [[ ${#PROMPTS[@]} -eq 0 ]]; then
+  # Two shapes: a short factual completion, and a longer instruction that keeps
+  # a reasoning model inside its channel for a while.
+  PROMPTS=(
+    "The capital of France is"
+    "List three prime numbers greater than 100 and explain briefly how you checked each one."
+  )
+fi
+
+if [[ -z "$OUT_DIR" ]]; then
+  OUT_DIR="$(mktemp -d -t ab_output_equality)"
+fi
+mkdir -p "$OUT_DIR"
+
+echo "baseline: $BASELINE_BIN"
+echo "arm:      $ARM_BIN"
+echo "tokens:   $MAX_TOKENS   (temp 0, --show-reasoning forced)"
+echo "outputs:  $OUT_DIR"
+echo
+
+# Run one generation and reduce it to just the generated text.
+#
+# Everything up to and including the last `Generating...` line is loader banner,
+# and the trailing bracketed lines are the timing block. The reasoning notice is
+# stripped too, though `--show-reasoning` means it cannot fire here. Redirecting
+# stdout to a file also makes `is_terminal()` false, so `--show-reasoning` emits
+# no dim SGR codes and the captured text is plain.
+run_one() {
+  local bin="$1" model="$2" prompt="$3" out="$4"
+  "$bin" generate -m "$model" -p "$prompt" -n "$MAX_TOKENS" --temp 0 --show-reasoning \
+    > "$out.raw" 2> "$out.err"
+  local status=$?
+  awk 'f { print } /^Generating\.\.\.$/ { f = 1 }' "$out.raw" \
+    | sed -E '/^\[Generated [0-9]+ tokens in /d; /^\[Profile Results\]$/d; /^\[All [0-9]+ generated tokens went to the reasoning channel;/d' \
+    | sed -e :a -e '/^\n*$/{$d;N;};/\n$/ba' \
+    > "$out"
+  return $status
+}
+
+fail=0
+inconclusive=0
+pair=0
+printf '%-52s %-44s %s\n' "MODEL" "PROMPT" "RESULT"
+for model in "${MODELS[@]}"; do
+  model_tag="$(basename "$model")"
+  for prompt in "${PROMPTS[@]}"; do
+    pair=$((pair + 1))
+    stem="$OUT_DIR/$(printf '%02d' "$pair")_${model_tag}"
+    prompt_tag="$(printf '%s' "$prompt" | cut -c1-40)"
+
+    if ! run_one "$BASELINE_BIN" "$model" "$prompt" "${stem}.baseline"; then
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "ERROR (baseline run failed, see ${stem}.baseline.err)"
+      fail=1
+      continue
+    fi
+    if ! run_one "$BASELINE_BIN" "$model" "$prompt" "${stem}.control"; then
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "ERROR (control run failed, see ${stem}.control.err)"
+      fail=1
+      continue
+    fi
+    if ! run_one "$ARM_BIN" "$model" "$prompt" "${stem}.arm"; then
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "ERROR (arm run failed, see ${stem}.arm.err)"
+      fail=1
+      continue
+    fi
+
+    if ! cmp -s "${stem}.baseline" "${stem}.control"; then
+      # The baseline does not reproduce itself, so nothing can be attributed to
+      # the arm on this pair. Say so rather than reporting a difference.
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "INCONCLUSIVE (baseline is not run-to-run stable)"
+      inconclusive=1
+      continue
+    fi
+    if cmp -s "${stem}.baseline" "${stem}.arm"; then
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "EQUAL"
+    else
+      printf '%-52s %-44s %s\n' "$model_tag" "$prompt_tag" "DIFFERS"
+      fail=1
+    fi
+  done
+done
+
+echo
+if [[ $fail -ne 0 ]]; then
+  echo "result: at least one pair differs or errored. Diff the saved files under $OUT_DIR."
+  exit 1
+fi
+if [[ $inconclusive -ne 0 ]]; then
+  echo "result: no differences, but at least one baseline was not stable against itself."
+  echo "        Those pairs prove nothing about the arm; pick a stable checkpoint or"
+  echo "        use the teacher-forced logit trace instead (docs/benchmarks.md)."
+  exit 2
+fi
+echo "result: every pair is byte-identical between baseline and arm."

src/commands/chat.rs

@@ -647,6 +647,11 @@ fn stream_turn<M: LanguageModel>(
     let mut generated_ids: Vec<u32> = Vec::with_capacity(max_tokens);
     let mut stdout = io::stdout();
     let dim = stdout.is_terminal();
+    // Whether anything ever reached the content channel. A turn that stays
+    // inside `<think>` to the end prints nothing at all, which reads as a hung
+    // or broken model rather than a suppressed channel, so the tail below says
+    // which one it was.
+    let mut saw_visible_text = false;
 
     session.generate_streaming(
         model,
@@ -659,6 +664,7 @@ fn stream_turn<M: LanguageModel>(
                 let visible =
                     reasoning_stream::render_visible(&filter.feed(&text), show_reasoning, dim);
                 if !visible.is_empty() {
+                    saw_visible_text |= !visible.trim().is_empty();
                     print!("{visible}");
                     let _ = stdout.flush();
                 }
@@ -674,12 +680,14 @@ fn stream_turn<M: LanguageModel>(
     if let Some(tail) = decode_state.flush(tokenizer) {
         let visible = reasoning_stream::render_visible(&filter.feed(&tail), show_reasoning, dim);
         if !visible.is_empty() {
+            saw_visible_text |= !visible.trim().is_empty();
             print!("{visible}");
             let _ = stdout.flush();
         }
     }
     let visible = reasoning_stream::render_visible(&filter.flush(), show_reasoning, dim);
     if !visible.is_empty() {
+        saw_visible_text |= !visible.trim().is_empty();
         print!("{visible}");
         let _ = stdout.flush();
     }
@@ -690,6 +698,17 @@ fn stream_turn<M: LanguageModel>(
     // not leak into the next turn's rendered history). Kept as the byte-exact
     // turn text used for the transcript.
     let reply = tokenizer.decode(&generated_ids, true).unwrap_or_default();
+
+    // The turn generated tokens but none of them left the reasoning channel, so
+    // nothing printed above. Say that, rather than leaving a blank turn.
+    if reasoning_stream::is_reasoning_only(&reply, saw_visible_text, show_reasoning) {
+        println!(
+            "[All {} generated tokens went to the reasoning channel; the content channel is empty. Restart with --show-reasoning to see them.]",
+            generated_ids.len()
+        );
+        println!();
+    }
+
     Ok(reply)
 }
 

src/commands/generate.rs

@@ -1113,17 +1113,32 @@ fn filter_reasoning_for_display(
     mlxcel::reasoning_stream::render_full(&markers, generated_text, primed, show_reasoning, dim)
 }
 
+/// Print the generation and its timing line.
+///
+/// `reasoning_only` comes from [`mlxcel::reasoning_stream::is_reasoning_only`]:
+/// the model generated normally but every token landed in the suppressed
+/// reasoning channel, so `generated_text` is empty here. Saying so is the whole
+/// point of the flag. A silent blank has twice been read as a broken model or a
+/// broken patch, once while it was the safety half of an A/B measurement.
 fn print_generation_result(
     generated_text: &str,
     stats: &GenerationStats,
     profile: bool,
+    reasoning_only: bool,
 ) -> Result<()> {
     print!("{}", generated_text);
     io::stdout().flush()?;
 
     println!();
     println!();
 
+    if reasoning_only {
+        println!(
+            "[All {} generated tokens went to the reasoning channel; the content channel is empty. Re-run with --show-reasoning to see them.]",
+            stats.generated_tokens
+        );
+    }
+
     if profile {
         println!("[Profile Results]");
         stats.print();
@@ -2496,7 +2511,12 @@ fn run_generate_once(mut args: GenerateArgs) -> Result<()> {
             &generated_text,
             args.generation.show_reasoning,
         );
-        print_generation_result(&visible, &stats, args.generation.profile)?;
+        let reasoning_only = mlxcel::reasoning_stream::is_reasoning_only(
+            &generated_text,
+            !visible.trim().is_empty(),
+            args.generation.show_reasoning,
+        );
+        print_generation_result(&visible, &stats, args.generation.profile, reasoning_only)?;
         mlxcel_core::clear_memory_cache();
         return Ok(());
     }
@@ -2635,7 +2655,12 @@ fn run_generate_once(mut args: GenerateArgs) -> Result<()> {
         &generated_text,
         args.generation.show_reasoning,
     );
-    print_generation_result(&visible, &stats, args.generation.profile)?;
+    let reasoning_only = mlxcel::reasoning_stream::is_reasoning_only(
+        &generated_text,
+        !visible.trim().is_empty(),
+        args.generation.show_reasoning,
+    );
+    print_generation_result(&visible, &stats, args.generation.profile, reasoning_only)?;
 
     // Cleanup
     mlxcel_core::clear_memory_cache();