[Feature] LoRa Training Support #2099

Feature Summary

Add support for in-library LoRA training for supported models

Detailed Description

Currently, stable-diffusion.cpp excels at fast, lightweight inference across various hardware backends (CUDA, Vulkan, Metal, etc.). However, training or fine-tuning models (such as training custom LoRAs) still requires heavy Python environments like diffusers or Kohya_ss.

I would like to request native support for LoRA training/fine-tuning within stable-diffusion.cpp. This would allow users to train lightweight style or character LoRAs efficiently on consumer hardware directly using C/C++ binaries, matching the performance and low memory footprint philosophy of the project.

Alternatives you considered

• Using Python-based tools like Kohya_ss or the Hugging Face Diffusers training scripts (which require heavy dependencies and are slower/more resource-heavy on consumer devices).
• Looking into whether llama.cpp's training/fine-tuning utilities can serve as an architectural reference.

Additional context

To make this feature robust and practical for consumer hardware, the CLI/tooling should support the following key parameters and features:

• LoRA Rank (dimension): Ability to configure the rank (e.g., r=4, 8, 16, 32) to balance model capacity and file size.
• Epochs & Repeats: Configurable training epochs and dataset repeat counts for controlling training duration and dataset exposure.
• Learning Rate: Adjustable learning rate with fine-grained control over weight updates.
• Resolution & Aspect Ratio (AR) Bucketing: Support for defining training resolutions and dynamic aspect ratio bucketing/ratios to train on mixed image dimensions without losing composition context through aggressive center-cropping.
• Learning Rate Scheduling: Support for different schedules including cosine, linear, and constant.
• Optimizations: Memory and performance optimizations (such as gradient accumulation or mixed precision) to fit within limited VRAM constraints.
• Checkpoint Restore: Automatic or manual checkpoint saving/restoring to recover seamlessly in case training is aborted or fails due to unexpected interruptions (like power loss).