Importance-Aware Low-Rank Distillation of Diffusion Transformers

To address the low deployment efficiency of Diffusion Transformer (DiTs), the research team proposed the SVDtrunc two-step block-level compression strategy. This approach first allocates and compresses model blocks with the lowest importance according to the global parameter budget, and then fine-tunes all blocks through modular knowledge distillation and modified flow objectives to restore performance. Experiments were conducted on the FLUX.dev model, testing compression from 40% to 90% of the original parameter count. The results showed that SVDtrunc outperformed existing competitive methods in the three benchmarks: GenEval, HPSv2, and DPG. In particular, it approached full performance while retaining 68% of the parameter budget and remained competitive with 57% of the parameter count. Additionally, this approach complements step distillation, achieving good results even without fine-tuning.