Repository navigation
Mage-Flow-Edit-Turbo ~1.6x slower on ROCm after graph-plan caching change (#1963) #2014
Description
Description
After bisecting a performance regression on Mage-Flow-Edit-Turbo (img-edit workflow) on ROCm/HIP, I found it's caused by commit 5ebce93 ("fix: reuse graph plans when scale parameters change", #1963). Sampling time for 16 steps at 1024×1024 goes from ~162s to ~266s (~1.64×) — output is pixel-identical (same seed), so this is purely a performance regression, not a correctness issue.
Notably ROCm-specific: The same test on the Vulkan backend shows no regression at all (154–156s before and after this commit).
Environment
- OS: Debian 13, AMD Ryzen AI Max+ 395 (Strix Halo), gfx1151, ROCm 7.2.4
- Build:
-DSD_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1151 -DCMAKE_BUILD_TYPE=Release
Bisect result
5ebce93342718ff7c0afa69b9b51d3fc0a1fd991 is the first bad commit
commit 5ebce93342718ff7c0afa69b9b51d3fc0a1fd991
fix: reuse graph plans when scale parameters change (#1963)
src/core/ggml_graph_cut.cpp | 16 ++++++++++++++--
Parent commit and everything before it: ~150–165s (matches the known-good baseline at 97d2990, three commits before the regression). 5ebce93 itself and everything after (tested up to current master c678dfe): ~230–266s.
Steps to reproduce
Weights: Mage-Flow-Edit-Turbo (Comfy-Org/Mage-Flow, bf16), Qwen3VL-4B-Instruct (llm + llm_vision), own VAE.
sd-cli \
--diffusion-model mage_flow_edit_turbo_bf16.safetensors \
--llm Qwen3VL-4B-Instruct-Q8_0.gguf \
--llm_vision mmproj-Qwen3VL-4B-Instruct-F16.gguf \
--vae mage_flow_vae_bf16.safetensors \
--sampling-method euler --steps 16 --cfg-scale 1.0 -W 1024 -H 1024 --diffusion-fa \
--ref-image <any image> \
-p "change the background to a beach at sunset" \
-o out.png
Hypothesis
The commit stops including op_params for GGML_OP_SCALE nodes in the graph-layout cache signature (intended to allow reusing a cached plan when only a scale value changes). For Mage-Flow-Edit-Turbo's graph, this may cause structurally different graphs to collide on the same cache key, forcing extra reconciliation work — apparently only costly on the HIP/ROCm backend path, not Vulkan. I haven't dug further into ggml_graph_cut.cpp internals to confirm the exact mechanism.
Additional notes
Related to (but distinct from) #2009 (SIGFPE in Wan video path) — same bisect investigation surfaced this as a second issue. Happy to test a patch.
Activity
leejet commentedon Sep 21, 2026
Thanks for the detailed bisect and the Vulkan comparison. I traced the change, but I haven't confirmed the root cause on ROCm yet.
One detail worth checking:
5ebce93changes the signature used by both the graph-cut plan cache and allocator reservation reuse. It still compares tensor shapes, strides, view offsets, and source connections, so I haven't found evidence of structurally different graphs colliding.I also ran a local graph-only check using Mage-Flow with BF16 weight metadata, one reference latent, flash attention enabled, and 16 different timesteps. The graph had 2,018 nodes, including 128 SCALE nodes. All SCALE parameters, graph-layout signatures, and binding signatures stayed identical across timesteps. Mage's timestep embedding is passed as tensor data, rather than a changing SCALE parameter. This suggests that this graph should already reuse its plan before the change. This was a CPU-side graph construction check without loading weights or running inference, so it does not reproduce or rule out the ROCm regression.
Could you repeat the same benchmark on both the parent commit and
5ebce93, withGGML_CUDA_DISABLE_GRAPHS=1set and--log-level debugadded? Despite its name, this environment variable also disables HIP graphs. Please keep the seed and all other arguments unchanged.The HIP backend has a separate graph capture/replay cache whose warmup depends on tensor metadata and addresses. If disabling HIP graphs removes the performance gap, that would narrow the investigation to this path. Otherwise, we can focus on allocation and execution differences. Logs from both versions would help, especially the occurrences of
CUDA graph warmup complete,CUDA graph warmup reset, andbuild cached graph cut plan done.basementops commentedon Sep 21, 2026
Thanks for the detailed follow-up. I ran the requested comparison, but the result is more surprising than expected — I want to report it honestly rather than send a misleading conclusion.
Setup: rebuilt separately at the parent commit (
7f986a9) and at5ebce93, both-DSD_HIPBLAS=ON -DAMDGPU_TARGETS=gfx1151, same test as the original report (Mage-Flow-Edit-Turbo, 1024×1024, 16 steps, euler, cfg 1.0,--diffusion-fa, seed 42).With
GGML_CUDA_DISABLE_GRAPHS=1 --log-level debug:7f986a9(parent)5ebce93No
CUDA graph warmup complete/resetlines in either log (expected, graphs disabled).build cached graph cut plan doneappears 5 times in both logs, identical pattern (mage_vae encode, 2× qwen3vl conditioning, mage_flow, mage_vae decode), each taking 0–4ms — plan is built once and reused across all 16mage_flow executing segmentsteps in both commits, matching your own graph-construction check.Then I went back and re-ran the control (graphs enabled, default) on these same fresh builds — and couldn't reproduce the original regression at all:
5ebce93, graphs enabled)CUDA graph warmup completedoes appear in these logs (confirming graphs were genuinely active), but sampling time stays in the same 145–160s range regardless — no sign of the ~1.64× slowdown under any of these four conditions.I don't have a solid explanation for the discrepancy with the original measurement. One thing worth mentioning for transparency: the original number was measured before a full host reboot (unrelated incident, a GPU memory/driver issue on our side), and everything I've tested since was measured after — I can't rule out that the original run was affected by some environmental state that a clean boot resets, though I also can't confirm that's what happened.
I'm hesitant to invest in a much more involved test (a persistent
sd-serverprocess handling two differently-shaped requests in sequence, to specifically try to trigger the cache-key collision you described) without more confidence it would actually reproduce something. Happy to try that if you think it's worth it, or if you have a more specific repro that differs from what I've done here — otherwise I'd currently lean toward this being an artifact of the original measurement rather than a real, reliably-reproducible regression. Sorry for the back-and-forth — wanted to give you the full picture rather than a tidy but possibly wrong answer.leejet commentedon Sep 25, 2026
Thanks for the thorough retesting. Since the slowdown no longer reproduces with HIP graphs either enabled or disabled, and both revisions reuse the graph plan as expected, I’m closing this as not reproducible for now.
The reboot may have cleared a transient GPU/driver state, but the cause remains unconfirmed. Please feel free to reopen if it happens again; a debug log and GPU/driver diagnostics captured before restarting would help. No further testing is needed right now.