Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized Diffusion Models

Authors: Tianjing Li, Wei Zhu

arXiv: 2610.10550 [cs.CL]

Submitted: 24 September 2026

DOI: https://doi.org/10.48550/arXiv.2610.10550

Abstract

Personalizing text-to-image diffusion models from a few reference images requires preserving subject identity while following prompts that describe new contexts. Full-model fine-tuning is parameter-intensive, whereas low-rank adaptation (LoRA) reduces the number of trainable parameters but leaves open how adaptation capacity should be distributed across layers.

We introduce Diffu-LoRA, a parameter-efficient method that learns this allocation through gated low-rank adaptation. Diffu-LoRA inserts trainable low-rank components into the linear layers of Transformer blocks and assigns a learnable gate to each component. Bilevel optimization updates the adaptation weights and gate parameters on separate data splits, while progressive pruning removes components with the lowest gate values to meet a prescribed rank budget. This procedure allocates adaptation capacity nonuniformly across layers while keeping the pretrained backbone frozen.

Experiments with Stable Diffusion on subjects from DreamBooth and additional collected datasets show improved overall subject fidelity and prompt alignment relative to the evaluated fine-tuning baselines. Ablation studies examine the contributions of bilevel optimization, progressive pruning, and adapter placement. These results support learned rank allocation as a practical approach to parameter-efficient diffusion model personalization.

Key Contributions

Gated low-rank adaptation: Trainable low-rank components are inserted into the linear layers of Transformer blocks, each paired with a learnable gate that controls its contribution.
Bilevel optimization: Adaptation weights and gate parameters are updated on separate data splits, enabling the model to learn where capacity should be concentrated.
Progressive pruning: Components with the lowest gate values are pruned to satisfy a prescribed rank budget, yielding a nonuniform allocation of adaptation capacity across layers.
Frozen backbone: The pretrained diffusion backbone remains fixed, keeping the method parameter-efficient.

Results

Evaluated on Stable Diffusion with subjects from DreamBooth and additional collected datasets, Diffu-LoRA improves overall subject fidelity and prompt alignment compared to the fine-tuning baselines tested. Ablation studies isolate the effects of bilevel optimization, progressive pruning, and adapter placement.

Subjects

Computation and Language (cs.CL)

Cite as: arXiv:2610.10550 [cs.CL]

Submission History

[v1] Thu, 24 Sep 2026 16:57:09 UTC (253 KB)

via ArXiv CL+LG

Diffu-LoRA: A Novel Low-Rank Adaptation for Personalized