[AlexCheema]

Builds on #2508 (MLX 0.32.4), now merged into main.

Summary

macOS releases the residency of wired Metal allocations once the GPU has been idle for a second or two. The first request after a pause then has to re-wire the model's weights before its prefill can run. The cost is roughly proportional to the weights: about 7 ms per GB on a Mac Studio (M3 Ultra), so seconds for a model of a few hundred GB. Every request that arrives after a short pause pays it: a chat user typing, or a benchmark with a cooldown between requests.

MLX 0.32.4 can re-request residency on a timer. MLX_METAL_RESIDENCY_REFRESH_INTERVAL_MS runs a renewal thread that submits no GPU work. The runner now sets it to 500 ms, next to MLX_METAL_FAST_SYNCH, before MLX creates its Metal device.

• A value already set in the environment is kept.
• 0 disables the renewal.

Measurements

Mac Studio (M3 Ultra, 512 GB). The wired limit is set as exo sets it (mx.set_wired_limit to the recommended working set). Each cell is a 512-token prefill after the given idle gap (each followed by a back-to-back control at ~191 ms).

8 real layers of GLM-5.3-4bit (30 GB of weights):

refresh interval	0 s gap	1 s	3 s	10 s	30 s
unset (today)	192 ms	204 ms	236 ms	443 ms	446 ms
500 ms	192 ms	204 ms	212 ms	220 ms	225 ms
1000 ms	191 ms	202 ms	215 ms	221 ms	225 ms
2000 ms	190 ms	206 ms	464 ms	222 ms	222 ms

• The ~30 ms left after a long gap with the refresh on is the GPU clocking back up.
• At 2000 ms, residency was already lost between renewals (the 3 s gap).
• Without mx.set_wired_limit the same gaps cost ~55 ms flat. The penalty comes from the wired allocations.

Control: Qwen3-1.7B-4bit (1 GB), 10 s gap: 137 ms unset vs 105 ms at 500 ms (back to back: 70-77 ms).

Full model: GLM-5.3-4bit (418 GB) on one M3 Ultra Mac Studio, 512-token requests through the API, 10 s cooldown between requests, 4 requests after a warm-up:

interval	TTFT median [min-max]	prefill tok/s	decode tok/s
0 (refresh off, the current behaviour)	8.45 s [8.36-9.28]	62.7	19.3
500 ms	2.90 s [2.89-2.92]	194.7	19.4

• TTFT is 2.9x lower; decode is unchanged.

Cost of the renewal:

• idle CPU of the runner process over 60 s: 0.005 % of one core unset, 0.018 % at 500 ms, 0.013 % at 1000 ms;
• steady-state decode unchanged (5.55 vs 5.56 ms/token on the 8-layer model).

Test plan

• uv run basedpyright, uv run ruff check, nix fmt, uv run pytest.
• New unit tests (test_residency_refresh.py):
  • the default interval;
  • an empty value;
  • a user value, including 0;
  • the runner's MLX environment setup (configure_mlx_environment) sets the variable and keeps a user's value.

🤖 Generated with Claude Code