ci: run the C++ and Python tests a second time on the CPU

[davidtai]

Proposed changes

mlx/backend/cpu has 45.89% line coverage in the Fork Tests report (run 36587584091). The C++ and Python tests run one time, with the GPU as the default device. So the CPU code of an op that also has a Metal implementation does not run. quantized.cpp, masked_mm.cpp, scan.cpp, softmax.cpp, logsumexp.cpp and fft.cpp are at 0%.

• A new step, "Run the tests on the CPU", runs after the GPU test step in the same job. It sets DEVICE=cpu. tests/tests.cpp and python/tests/mlx_tests.py both read DEVICE and set the CPU as the default device.

• The step runs the C++ tests (./build/tests/tests) one time. Then it runs one process for each Python test file, three at a time (the runner has 3 cores). Each file runs in a child process that writes its own log, so a crash stops only that file. The step prints each log in its own group, and one result line for each file.

• The step leaves out two tests on the CPU, by a named list (CPU_EXCLUDE) in the workflow. They take more than 5 minutes on the CPU. The GPU pass still runs them:

  • test_fast_sdpa.TestFastSDPA.test_sdpa_long_masked_sequence: 21 min 36 s on the CPU.
  • test_fast_sdpa.TestFastSDPA.test_sdpa_full_head_dim_256: 6 min 46 s on the CPU.
    Nothing else is left out. The tests that fail on the CPU run and show as failures.

• The step writes its profiles to the same profiles folder, with the name pattern cpu-%p-%m.profraw. The report step merges all profiles, so one report covers the two runs.

• The step fails when a test fails. A failure makes the job red. The report step still runs after a failure, as it does after a GPU test failure.

• The job summary shows the result of each test step, and one new sentence says that the tests run two times.

• Two Python tests check behavior that exists only in the Metal backend. They skipped only when Metal is not available, so on this macOS runner they ran on the CPU and failed. Each now also skips when the default device is not the GPU. The GPU pass still runs every assertion:

  • test_conv.TestConv.test_conv2d_winograd_batch_tiling checks that a working-set budget too small for one batch element makes the Metal conv fall back from Winograd to implicit GEMM, so the output differs bit for bit from the Winograd output. The CPU backend has no Winograd path: MLX_CONV_WINOGRAD_WORKING_SET and MLX_CONV_WINOGRAD_TILE_BATCH are read only in mlx/backend/metal/conv.cpp:75 and :99. On the CPU every run is the same conv, so "differs from untiled" cannot hold. Not a CPU defect.
  • test_fft.TestFFT.test_fft_too_large checks that the Metal four-step FFT plan throws for 2**25 values (mlx/backend/metal/fft.cpp:115, :148). The CPU FFT uses pocketfft (mlx/backend/cpu/fft.cpp) and has no size limit, so it gives a result. Not a CPU defect.
  • Upstream ml-explore/mlx has the same two tests with the same mx.metal.is_available() guard. Upstream runs DEVICE=cpu only on Linux and Windows, where Metal is not available, so the guard skips them there.

• fork.yaml: test_conv.py and test_fft.py move from the upstream v0.32.2 sync section (which says its files are byte-identical to upstream) to "Fork tooling", with one sentence on the CPU pass and the two skips.

The distributed tests and the example extension do not run a second time. No library code changes.

Results

Fork Tests run on head ed2e48f: https://github.com/Layr-Labs/mlx/actions/runs/36778058130 (success). Before is #29's run 36587584091 on head 02720cc. Values are from the TOTAL rows in the step log.

Row	Lines before	Lines after	Branches before	Branches after	Functions before	Functions after
All of mlx/	70.08%	80.52%	62.69%	69.99%	72.64%	79.72%
mlx/backend/cpu	45.89%	92.80%	39.21%	81.42%	56.80%	97.28%
mlx/backend/metal	81.34%	81.34%	66.62%	66.62%	82.24%	82.24%
mlx/backend/common	89.37%	90.90%	84.03%	85.87%	93.04%	95.65%

The denominator does not change (50737 lines).

• Tests on the GPU: C++ 290 of 290 passed. Python Ran 875 tests ... OK (skipped=52). test_conv2d_winograd_batch_tiling and test_fft_too_large ran and passed.
• Tests on the CPU: C++ 290 of 290 passed. Python 33 of 33 test files pass, 872 tests ran. The two tests above show as skipped with their reason.

Earlier run 36752129570 (head 8aadfc1, attempt 2) had the two CPU failures that this head fixes. Its attempt 1 was cancelled at the 60-minute limit in the GPU test step, which this PR does not change: the GPU Python tests printed their result and then the process did not exit for 41 minutes.

Test plan

• actionlint .github/workflows/fork-tests.yml: no findings.
• /usr/bin/python3 scripts/check_forkdiff.py --upstream-ref refs/remotes/upstream/main: passes.
• The Fork Tests run on this head is the test.

CONTRIBUTING checklist

• Formatting: actionlint passes on the workflow; the Python change is two guard lines per test.
• Tests: no library code; two test guards change; the Fork Tests run of this PR is the test.
• Benchmarks: not applicable, no library code changes.

Dependencies

Merge this PR after #28 and #29. This branch is based on ci/coverage-scope (#29's head, 02720cc), which is based on #28. The PR base is ci/coverage-scope, so the diff shows only this change.

Authorship

Written by Claude (Opus 5.5) under David Tai's direction.

• ☐ I understand it is strictly prohibited to use AI to write PR description
• AI usage disclosure: this change and this description were written by Claude (Opus 5.5) under David Tai's direction.

🤖 Generated with Claude Code