Diffusion models

Setup

• $q(y)$: true sampling distribution of observed data
• $p_{\theta }(y)$: learned sampling distribution
• ${\text{x}}^{1:T}$: noisy latent variables at each step $t$

Forward process

In the forward process, noise is continuously added to the data across multiple steps, converging toward white noise ${\text{x}}^T\sim N(0,1)$: $q({\text{x}}^t|{\text{x}}^{t-1}):=N(\sqrt{1-{\beta }_t}{\text{x}}_{t-1},{\beta }_t\text{I})$ The fixed Gaussian forward process means we can directly calculate/sample from $q({\text{x}}^t|y)$ without calculating all the intermediate steps (similar to Autoregressive models): ${\text{x}}^t=\sqrt{{\bar{\alpha }}_t}y+\sqrt{(1-{\bar{\alpha }}_t)}ϵ$ Where ${\alpha }_t=1-{\beta }_t$, ${\bar{\alpha }}_t={\prod }_{i=1}^t{\alpha }_i$, and $ϵ$ is white noise. Again, this is 100% analogous to the forecasting equation for an AR(1).

In practice, ${\alpha }_t$ is chosen to be close to 1, as this yields the best results.

Denoising

In the denoising or reverse diffusion process, noise is progressively removed over multiple steps, modeling the inverse process: $p_{\theta }({\text{x}}^{t-1}|{\text{x}}^t):=N({\mu }_{\theta }({\text{x}}^t,t),{\sigma }_t\text{I})$ Where ${\sigma }_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}{\beta }_t$.

${\mu }_{\theta }$ is parameterized using a denoising network, ${ϵ}_{\theta }$ (of which many exist, including the popular U-Net): ${\mu }_{\theta }({\text{x}}^t,t)=\frac{1}{\sqrt{{\alpha }_t}}({\text{x}}^t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }({\text{x}}^t,t))$ This model is trained using the objective function: $E_{y,ϵ,t}[||{ϵ}_{\theta }({\text{x}}^t,t)-ϵ|{|}^2]$

---

References

@kolloviehPredictRefineSynthesize2023

@harveyFlexibleDiffusionModeling2022

eonline