Problem / Background
scripts/ab_output_equality.sh is the output-equality half of an A/B: it runs the baseline and arm binaries over the same (model, prompt) pairs at --temp 0 --show-reasoning and requires byte-identical generated text. It validates that --baseline and --arm are not the same path (scripts/ab_output_equality.sh:78-82), never that they differ in content, so two paths holding identical bytes pass and every pair reports EQUAL for comparing the unchanged binary against itself.
That is the form the mistake actually takes: the recipe in the script's own header says to build the baseline, copy the binary aside, patch, and rebuild. A cargo build that no-ops leaves the copy and the "rebuilt" arm identical. A bridge .cpp edit produced that 0.12s no-op on an M5 Max on 2026-09-09 while an M1 Ultra rebuilt the same edit normally, so whether the gate is honest depends on the host it runs on.
Second defect, same file: usage() prints the header through a hard-coded sed -n '2,45p' window (scripts/ab_output_equality.sh:53-54), so a header one line past it silently truncates the documented exit statuses out of --help.
Proposed Solution
Refuse byte-identical arms by default (cmp -s, before any generation runs), with an explicit opt-out for a source change that genuinely compiles to the same output, and make the usage window track the real header length instead of a fixed range.
Acceptance Criteria
Verification
bash -n scripts/ab_output_equality.sh && shellcheck scripts/ab_output_equality.sh
/bin/cp target/release/mlxcel /tmp/mlxcel.copy
./scripts/ab_output_equality.sh --baseline /tmp/mlxcel.copy --arm target/release/mlxcel --model models/mlx/granite-4.0-h-tiny-4bit; echo "exit=$?"
./scripts/ab_output_equality.sh --help | tail -5
Technical Considerations
The Metal half of the last acceptance criterion of #1803 is the next user of this gate; that change touches src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp and src/lib/mlx-cpp/turbo/*.cpp, the same bridge-.cpp class that triggered the no-op. Refs #1803
Problem / Background
scripts/ab_output_equality.shis the output-equality half of an A/B: it runs the baseline and arm binaries over the same (model, prompt) pairs at--temp 0 --show-reasoningand requires byte-identical generated text. It validates that--baselineand--armare not the same path (scripts/ab_output_equality.sh:78-82), never that they differ in content, so two paths holding identical bytes pass and every pair reports EQUAL for comparing the unchanged binary against itself.That is the form the mistake actually takes: the recipe in the script's own header says to build the baseline, copy the binary aside, patch, and rebuild. A
cargo buildthat no-ops leaves the copy and the "rebuilt" arm identical. A bridge.cppedit produced that 0.12s no-op on an M5 Max on 2026-09-09 while an M1 Ultra rebuilt the same edit normally, so whether the gate is honest depends on the host it runs on.Second defect, same file:
usage()prints the header through a hard-codedsed -n '2,45p'window (scripts/ab_output_equality.sh:53-54), so a header one line past it silently truncates the documented exit statuses out of--help.Proposed Solution
Refuse byte-identical arms by default (
cmp -s, before any generation runs), with an explicit opt-out for a source change that genuinely compiles to the same output, and make the usage window track the real header length instead of a fixed range.Acceptance Criteria
--helprenders every header line, exit statuses included, with no hard-coded range inusage().Verification
Technical Considerations
The Metal half of the last acceptance criterion of #1803 is the next user of this gate; that change touches
src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cppandsrc/lib/mlx-cpp/turbo/*.cpp, the same bridge-.cppclass that triggered the no-op. Refs #1803