Repository navigation
Conversation
Use native multi-axis RoPE and fusion-friendly graph layouts for Krea2, retain reusable Vulkan graph allocations, streamline model uploads, and route the single-frame VAE through the optimized 2D paths. Pin the companion ggml-vulkan implementation that provides the tuned cooperative kernels and fused dispatches.
Default server-side PNG encoding to the measured level 4 setting, expose a validated command-line override, and reserve the exact base64 output capacity. File and CLI encoding behavior outside the server remains unchanged.
|
That's way too many barely related changes for a single PR. If you want these changes to get approved, It would be better to split this into multiple separate PRs that can be reviewed independantly, as per the contribution guidelines. You could use PR stacking maybe. |
|
@stduhpf I've already opened three split PRs for the GGML changes in the llama.cpp repo, I don't think I can reasonably split the sd.cpp changes themselves. |
|
I mean some changes to sd.cpp don't seem related to the rest, like the PNG compression level stuff for example, it doesn't look like it absolutely needs to be bundled with the ggml graph optimizations |
|
Yeah, fair enough. I can cut out the less obvious ones. |
Piotr Wilkin's PR adapted to the post-split tree (upstream leejet#1945 split ggml_extend.hpp; the pipeline refactor moved StableDiffusionGGML into src/pipeline/): - krea2: native MROPE path (gen_krea2_rope_data + ggml_rope_multi) replaces the precomputed cos/sin pe matrix; swiglu_split in the MLP - param transforms: load-time +1 fold of RMSNorm scales, keyed by ModelComponent instead of the removed desc string; registered from build_diffusion_runners; writable mmap also enabled for krea2 - ggml_ext_attention_ext: keep strided v cast on Vulkan - model_loader: per-backend-buffer mutexes for parallel tensor upload - server: --png-compression-level option, base64 reserve - runner context: graph + param_transform_registered fields (ctx->gf from the PR maps to the existing ctx->graph) ggml submodule kept at 89c4413 (upstream master, hip INT8 tensor matmul/convrot) instead of the PR's 76f89d6 branch: the two set_prec APIs the PR needs already exist there and the PR's extra commits are Vulkan-RDNA3 kernels, irrelevant on HIP. The PR's compute-buffer retention flag is obsolete: the split runner already keeps the workspace across steps and frees it at runner_end.
|
Would ya'll mind unclogging this PR? There's a lot of performance to gain on various hardware, with some focus. And this model is great for 12-16GB setups as it's 13B size makes it extremely forgiving to Q4 quanting of the model. |
3 participants
Summary
People in the Lemonade community have been complaining that sd.cpp is very slow compared to native PyTorch inference and a new backend (TheNoise) has been created to obtain fast generation, so I decided to try to match parity.
Related Issue / Discussion
Lots of kernel (see linked GGML fork/PR) and Krea changes that in total bring the generation time for a 1024x768 image in 8 steps on Krea2 Turbo Q8_0 on a Strix Halo from 77 seconds to 29 seconds.
Additional Information
Images are not bit-identical, but coherence/detail has been preserved at a similar level.
Note: this is probably NOT a one-time effort, I chose Krea2 first because that was chosen for TheNoise as one of the first supported models, but I want to do a sweep across a couple of the new supported image models to bring performance up to par and hopefully people with different hardware can use this to tune optimizations for their own hardware as well.
Checklist