Recovering Diversity in Distilled Diffusion Models via Low-Rank Policy-Optimized Merging

Timestep distillation accelerates diffusion models but often reduces sample diversity. We propose a low-rank policy-optimized model merging framework that combines a full-step teacher’s diversity with a distilled student’s efficiency. The method decomposes teacher-student weight differences into low-rank components and optimizes their merging coefficients using Group Relative Policy Optimization (GRPO), with image quality and diversity as rewards. Requiring only a single text prompt for optimization, the framework recovers diversity while maintaining or improving image quality and preference at the same inference cost, as demonstrated on SDXL and SD3.5-L.

Accepted to WACV 2027.

Authors: Hoigi Seo, Woojae Han, Joohoon Lee and Se Young Chun.