Skip to content

fix: the A/B output-equality gate accepts a baseline and arm that are byte-identical #1866

Description

@inureyes

Problem / Background

scripts/ab_output_equality.sh is the output-equality half of an A/B: it runs the baseline and arm binaries over the same (model, prompt) pairs at --temp 0 --show-reasoning and requires byte-identical generated text. It validates that --baseline and --arm are not the same path (scripts/ab_output_equality.sh:78-82), never that they differ in content, so two paths holding identical bytes pass and every pair reports EQUAL for comparing the unchanged binary against itself.

That is the form the mistake actually takes: the recipe in the script's own header says to build the baseline, copy the binary aside, patch, and rebuild. A cargo build that no-ops leaves the copy and the "rebuilt" arm identical. A bridge .cpp edit produced that 0.12s no-op on an M5 Max on 2026-09-09 while an M1 Ultra rebuilt the same edit normally, so whether the gate is honest depends on the host it runs on.

Second defect, same file: usage() prints the header through a hard-coded sed -n '2,45p' window (scripts/ab_output_equality.sh:53-54), so a header one line past it silently truncates the documented exit statuses out of --help.

Proposed Solution

Refuse byte-identical arms by default (cmp -s, before any generation runs), with an explicit opt-out for a source change that genuinely compiles to the same output, and make the usage window track the real header length instead of a fixed range.

Acceptance Criteria

  • Two paths with identical bytes exit non-zero before any model loads, naming the likely cargo no-op.
  • The opt-out flag proceeds to the runs; the existing same-path check still rejects.
  • --help renders every header line, exit statuses included, with no hard-coded range in usage().

Verification

bash -n scripts/ab_output_equality.sh && shellcheck scripts/ab_output_equality.sh
/bin/cp target/release/mlxcel /tmp/mlxcel.copy
./scripts/ab_output_equality.sh --baseline /tmp/mlxcel.copy --arm target/release/mlxcel --model models/mlx/granite-4.0-h-tiny-4bit; echo "exit=$?"
./scripts/ab_output_equality.sh --help | tail -5

Technical Considerations

The Metal half of the last acceptance criterion of #1803 is the next user of this gate; that change touches src/lib/mlxcel-core/cpp/mlx_cxx_kernels.cpp and src/lib/mlx-cpp/turbo/*.cpp, the same bridge-.cpp class that triggered the no-op. Refs #1803

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    area:benchmarkBenchmark harness and performance measurement (bench_*.sh, /update-benchmarks)priority:mediumMedium prioritystatus:reviewUnder reviewtype:bugBug fixes, error corrections, or issue resolutions

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions