[0xShug0]

Adds experimental CPU/GPU expert execution for disk-backed MoE sessions through the common model session option moe_cpu_share. The CLI exposes --moe-cpu-share as shorthand for --session-option moe_cpu_share=…; server configuration sets the value inside each model’s session_options. Cache hits remain on GPU; a configurable fraction of subsequent cache misses executes on CPU while GPU expert work proceeds. CPU computation uses the configured thread count and native quantized weights.

The default is 0, leaving hybrid execution disabled. The policy is separate from BackendConfig; backend selection and server-wide settings are unchanged from main. No model-local changes are required for eligible shared MoE callers.

For example, add "moe_cpu_share": 0.5 to the selected model’s session_options object alongside its existing disk-expert options. Recipes use --session-option <role>.moe_cpu_share=0.5; workflows may configure the option per step.

Implementation and scope

• Add explicit graph-region replacement to PreparedGraph and the GGML scheduler, preserving the original allocation graph and enforcing region boundaries around backend optimization.
• Add session-owned CPU expert graphs, reusable buffers, a persistent worker and cache-aware placement. Resident GPU hits execute before misses to avoid evicting pending hits.
• Shared code is also refactored: ordinary scheduler submission/callback execution uses an extracted helper, and graph preparation/cleanup checks for regions. Ordinary scheduling uses a separate template specialization without per-node region processing. These are shared-path changes, despite hybrid execution being opt-in.
• CPU-only execution and resident expert banks retain their normal execution paths. Unsupported registered regions fail explicitly.

Hybrid execution performance

RTX 5090, Ryzen 9 7900X, CUDA Debug build, eight CPU threads, disk experts enabled and eight GPU expert-cache slots. Same binary, moe_cpu_share=0 versus 0.5, measured sequentially in warmed A→B→B→A order. Wall time is the mean total for three measured requests per block, after a separate short warmup.

Model	Default wall time	Hybrid wall time	Time reduction	Peak VRAM, default → hybrid
Nemotron Transcribe	14.385 s	11.336 s	21.2%	6,473 → 6,483 MiB
Drama3	39.742 s	40.064 s	No meaningful change	6,599 → 6,645 MiB
Ming3 Audio	3.472 s	3.470 s	No meaningful change	4,151 → 4,189 MiB

Nemotron improves by 21.1% and 21.3% in the two run orders. Drama3 and Ming3 timing differences are consistent with run variation, rather than an established performance regression. Peak VRAM is the maximum process-attributed usage over each session's request sequence.

Default-path validation against main

Compared main 33ca47fcb against PR b0d9479f5 with moe_cpu_share omitted, using separate statically linked Debug binaries and identical build settings. Sequential warmed main→PR→PR→main runs used eight threads and local NVMe storage, with a separate short warmup before measured requests. The three MoE models ran on CUDA; FireRed VAD used CPU+BLAS to exercise the shared scheduler without MoE.

Model/path	Main RTF	PR RTF	Peak VRAM, main → PR
Drama3 disk experts	0.70483	0.69653	6,591 → 6,595 MiB
Drama3 full GPU	0.16362	0.16438	23,323 → 23,323 MiB
Nemotron Transcribe disk experts	0.25965	0.25560	6,473 → 6,473 MiB
Nemotron Transcribe full GPU	0.01798	0.01797	22,255 → 22,255 MiB
Ming3 Audio ASR disk experts	0.06200	0.06163	4,151 → 4,151 MiB
Ming3 Audio ASR full GPU	0.01526	0.01491	7,583 → 7,583 MiB
FireRed VAD CPU+BLAS scheduler	0.02610	0.02622	CPU inference

No material default-path slowdown was observed in this matrix. Drama3 full-GPU and the CPU control averaged approximately 0.5% slower; repeated paired comparisons changed sign. These small differences are not claimed as optimization gains or proof of literally zero overhead.

• 100 paired main/PR results match exactly after excluding timing, including transcripts, timestamps, speaker labels, confidence values and VAD outputs.
• All 30 Drama3 WAV pairs and six streamed-PCM pairs match exactly.
• Repeated requests and long→short session reuse match main's behavior.
• Logged buffer/workspace allocation sequences match. Warmed peak VRAM matches except one additional 4 MiB Drama3 disk-expert PR sample; the other PR block matches main.
• The comparison covers Drama3 default/perf-off TTS, Nemotron ASR with diarization/timestamps, Ming3 Audio ASR, and the CPU scheduler control. Other backends and Ming3 S2S are not claimed as covered.

Hybrid correctness and limitations

• Nemotron's six paired measured results match exactly after excluding timing, including transcripts, timestamps, speaker labels and confidence values.
• Ming3's transcript words match; one sentence-boundary space differs in the longer request.
• Repeated requests complete successfully. Drama3 repeats produce identical WAVs within each arm.
• Hybrid execution is not numerically parity-preserving. A short Drama3 test produces unrelated speech with hybrid enabled, while the default produces the requested sentence. This remains a quality limitation of the experimental option.
• Drama3's longer hybrid output lowers its RTF without reducing wall time; that is not counted as a speedup.

Debug CLI/server builds and the focused PreparedGraph and CUDA disk-expert tests pass. A real CUDA server run with separate ordinary and hybrid model entries returns matching transcripts; assignment counters confirm CPU expert execution only for the enabled path. The CLI shorthand also completes a real CUDA request with CPU expert assignments. Invalid values and session isolation are covered by the focused tests. git diff --check passes.