Diffusion models

Setup

• $q(y)$: true sampling distribution of observed data
• $p_{\theta }(y)$: learned sampling distribution
• ${\text{x}}^{1:T}$: noisy latent variables at each step $t$

Forward process

In the forward process, noise is continuously added to the data across multiple steps, converging toward white noise ${\text{x}}^T\sim N(0,1)$: $q({\text{x}}^t|{\text{x}}^{t-1}):=N(\sqrt{1-{\beta }_t}{\text{x}}_{t-1},{\beta }_t\text{I})$ The fixed Gaussian forward process means we can directly calculate/sample from $q({\text{x}}^t|y)$ without calculating all the intermediate steps (similar to Autoregressive models): ${\text{x}}^t=\sqrt{{\bar{\alpha }}_t}y+\sqrt{(1-{\bar{\alpha }}_t)}ϵ$ Where ${\alpha }_t=1-{\beta }_t$, ${\bar{\alpha }}_t={\prod }_{i=1}^t{\alpha }_i$, and $ϵ$ is white noise. Again, this is 100% analogous to the forecasting equation for an AR(1).

In practice, ${\alpha }_t$ is chosen to be close to 1, as this yields the best results.

Denoising

In the denoising or reverse diffusion process, noise is progressively removed over multiple steps, modeling the inverse process: $p_{\theta }({\text{x}}^{t-1}|{\text{x}}^t):=N({\mu }_{\theta }({\text{x}}^t,t),{\sigma }_t\text{I})$ Where ${\sigma }_t=\frac{1-{\bar{\alpha }}_{t-1}}{1-{\bar{\alpha }}_t}{\beta }_t$.

${\mu }_{\theta }$ is parameterized using a denoising network, ${ϵ}_{\theta }$ (of which many exist, including the popular U-Net): ${\mu }_{\theta }({\text{x}}^t,t)=\frac{1}{\sqrt{{\alpha }_t}}({\text{x}}^t-\frac{{\beta }_t}{\sqrt{1-{\bar{\alpha }}_t}}{ϵ}_{\theta }({\text{x}}^t,t))$ This model is trained using the objective function: $E_{y,ϵ,t}[||{ϵ}_{\theta }({\text{x}}^t,t)-ϵ|{|}^2]$

Intuition

One intuitive way of thinking about diffusion models that I came up with while watching Advancing Diffusion Models for Text Generation – think about diffusion modeling as picking up a handful of sand (sampled random noise) and throwing the grains of sand against a canvas (manifold of the data distribution you care about, e.g. images). If you throw the sand in a particularly skillful way, the patterns that end up on the canvas will look like recognizable images. The training process in diffusion models is essentially learning this motion of skillfully throwing a bunch of sand so as to end up with an interesting pattern on the canvas. This is obviously somewhat complicated and a bit of miracle that this even works / is possible!

---

References

@kolloviehPredictRefineSynthesize2023

@harveyFlexibleDiffusionModeling2022

https://www.youtube.com/watch?v=klW65MWJ1PY

eonline