Full-step teacher50 steps
ACCEPTED AT WACV 2027SDXL · SD3.5-L
Recovering Diversity in
Distilled Diffusion Modelsvia Low-Rank Policy-Optimized Merging
Teacher–student model merging for few-step diffusion
SDXL / DIVERSITY COMPARISON“A photo of a young woman with freckles”
01 / 02
Distilled student4 steps
Our merge ↗4 steps
SDXL figure samples. Nine generations per method; 50 steps for the teacher, 4 steps for the student and merged model.
01 — OVERVIEW
Recovering diversity after distillation
Timestep distillation makes diffusion models fast. But adversarial objectives can concentrate their outputs, trading away diversity. We recover much of that lost diversity by merging a distilled student with its full-step teacher.
Our framework decomposes their weight difference into low-rank directions, then learns how much of each direction to merge through Group Relative Policy Optimization (GRPO). Quality and diversity guide the merge, using just one calibration prompt.
4 steps
Same inference cost
as the distilled student
+84%
SDXL DINO diversity
relative to SDXL-turbo
1 prompt
“a photo of a cat”
for optimization
Diversity: 0.1064 → 0.1962 on GenEval (DINOv3); +84.4%, rounded. Results from Table 2.
02 — DIVERSITY COMPARISON
Sample diversity
SDXL teacher, distilled student, and merged model.
Select a prompt and compare nine samples.
SAMPLE 01 / 09
“A photo of a winter coat”
SDXL samples from the supplied main and supplementary figure assets. All displayed samples are 512 × 512 pixels. The viewer displays pre-generated images.
Compare all merging baselines +
Full nine-sample grids for each method.
03 — METHOD
Low-rank policy-optimized merging
Optimize the merge in weight space,
rather than backpropagating through generation.
LOW-RANK WEIGHT SPACE
Instead of one global interpolation coefficient, assign a separate coefficient to each singular-vector pair. This makes merging more expressive while keeping its representation compact.
Wmerged = Wstudent + Σk τk ãkb̃k⊤
04 — QUANTITATIVE RESULTS
Quantitative comparison
Evaluation on SDXL and SD3.5-L.
Select a model family to view Table 2.
Diversity DINOv3 ↑
OUR MERGE VS. STUDENT+84.4%
higher DINO diversity at the same 4-step inference cost.
DeQA quality ↑4.2414 → 4.3483
| Method | NFE | DINO ↑ | CLIP ↑ | DeQA ↑ | MIQA ↑ | FID ↓ | ImgR ↑ | PicS ↑ |
|---|
Diversity: 553 GenEval prompts, five images each. Quality and preference: COCO-10K. ↑ higher is better; ↓ lower is better. Values reproduced from Table 2. Gains vary by metric; our merge does not lead every column.
05 — QUALITATIVE RESULTS
Image quality comparison
Same seeds across methods.
Actual qualitative comparisons from Figure 4.
“a baseball player bunting at a baseball on a field.”
The paper reports improved crispness and reduced artifacts relative to competing merges. Images are extracted directly from the original figure assets corresponding to Figure 4.
06 — ABSTRACT
Abstract
Diffusion models enable high-fidelity, diverse image synthesis but incur high deployment cost, motivating timestep distillation. To mitigate blurriness, distillation methods widely adopt adversarial objectives, which frequently reduce sample diversity. Existing merging methods can raise diversity, but only at the cost of quality. To recover diversity without this quality penalty, we propose a low-rank policy-optimized model merging framework that combines the teacher’s diversity with the student’s efficiency.
Specifically, we decompose the teacher–student weight difference matrices via low-rank decomposition and optimize the coefficients of the resulting singular vectors, allowing finer-grained and memory-efficient merging. However, the student’s iterative few-step inference makes direct backpropagation through sampling intractable. We address this by modeling the interpolation coefficient logits as a Gaussian distribution policy and optimizing them via Group Relative Policy Optimization (GRPO), using image quality and diversity as reward signals.
Notably, this formulation remains highly data-efficient, requiring only a single text prompt for optimization. In contrast to prior merging methods, our method recovers much of the diversity lost to distillation while maintaining or improving quality and preference, achieving the best quality-diversity balance among merging approaches at the same inference cost. We demonstrate this on both SDXL and SD3.5-L, which differ in architecture and training objective.
07 — CITATION
BibTeX
@inproceedings{seo2027recovering,
title={Recovering Diversity in Distilled Diffusion Models
via Low-Rank Policy-Optimized Merging},
author={Seo, Hoigi and Han, Woojae and Lee, Joohoon
and Chun, Se Young},
booktitle={Proceedings of the IEEE/CVF Winter Conference
on Applications of Computer Vision (WACV)},
year={2027}
}Accepted at WACV 2027. Proceedings details will be updated when available.