Facilitated Diffusion Models (2023)

Understand the fundamentals of probabilistic noise reduction diffusion models

1.Introduction

In the recent past, I've talked about GAN and VAE as two important generative models that have been very successful and recognized. GANs work great for many applications; however, they are difficult to train and their output lacks diversity due to various challenges such as mode collapse and disappearing gradients, to name a few. Although VAEs have the strongest theoretical basis, modeling a good loss function is a challenge in VAEs that makes their output suboptimal.

There is another set of techniques that originate from probabilistic methods of likelihood estimation and are inspired by physical phenomena; is called Diffusion Models. The core idea behind Diffusion Models comes from the thermodynamics of gas molecules, where molecules diffuse from areas of high density to areas of low density. This movement is often referred to in the physics literature as entropy increase or heat death. In information theory, this amounts to the loss of information due to the gradual intervention of noise.

The key concept in diffusion modeling is that if we could build a learning model that could learn the systematic decay of information due to noise, it would be possible to reverse the process and thus recover the information from the noise. This concept is similar to VAE in that it tries to optimize an objective function by first projecting the data into latent space and then recovering it to the initial state. However, instead of learning the data distribution, the system tries to model a series of noise distributions in aMarkov chainand "decodes" the data by de-noising/de-noising the data in a hierarchical fashion.

2. Noise Reduction Diffusion Model

The idea of ​​removing noise from the diffusion model has been around for a long time. It has its roots in the concept of diffusion maps, which is one of the dimensionality reduction techniques used in machine learning literature. It also borrows concepts from probabilistic methods such asMarkov chainswhich has been used in many applications. The original Denoising Diffusion method was proposed inSohl-Dickstein et al.. [1].

A diffusion model of noise reduction is a two-step process: the forward diffusion process and the reverse or reconstruction process. In the forward diffusion process, Gaussian noise is successively introduced until the data becomes just noise. The inverse reconstruction process is done by learning the conditional probability densities using a neural network model. An example representation of this process can be seen in Figure 1.

3. referral process

We can formally define the process of direct diffusion as aMarkov chainand therefore, unlike a coder in the VAEs, does not require training. Starting with the initial data point, we add Gaussian noise toTsuccessive steps and obtain a set of noisy samples. Predicting probability density over timetdepends only on the immediate ancestor at the momentt-1and therefore the conditional probability density can be calculated as follows:

See Also

[理解论文]扩散模型在图像合成方面优于GAN

The complete distribution of the entire process can be calculated as follows:

Here, the mean and variance of the density function depend on a parameter βτ, which is a hyperparameter whose value can be taken as constant throughout the process or can be gradually changed in successive steps. For a differential parameter value assignment, there may be a function range that can be used to model the behavior (eg, sigmoid, tanh, linear, etc.).

However, the above derivation is sufficient to predict successive states if we wish to sample over any time interval.twithout going through all the intermediate steps, thus allowing for an efficient implementation, then we can reformulate the above equation by replacing the hyperparameter as ατ = 1 — βτ.Rephrasing what was said above becomes:

To produce samples in one time steptwith probability density estimate available in the time intervalt-1,we can use another concept from thermodynamics called, 'Langevin dynamics. According toLangevin dynamic stochastic gradient[2] we can sample the new states of the system just by the gradient of the density function in aMarkov chainupdates Sampling one new data point at a timetfor a step sizemibased on an earlier point in timet-1then it can be calculated as follows:

4. Reconstruction

The inverse process requires estimating the probability density at an earlier time step given the current state of the system. That means appreciating theq(hkt-1 | hht) whent=T and thus generating samples of isotropic Gaussian noise data. However, unlike the direct process, estimating the previous state from the current state requires knowledge of all past gradients which we cannot obtain without having a learning model that can predict such estimates. Therefore, we will have to train a neural network model that estimates therth(χτ-1|χτ) based on learned weightseuand the current state at the momentt. This can be estimated as follows:

The parameterization of the mean function was proposed byLo. and others[3] and can be calculated as follows:

The authors inLo. and others[3] suggested using a fixed variance function like Σθ = βτ. The sample at the momentt-1then it can be calculated as follows:

5. Training and Results

5.1. Model Building

The model used in the broadcast training model follows patterns similar to a VAE network; however, it is often kept much simpler and more straightforward compared to other network architectures. The input tier has the same input size as the data dimensions. There can be multiple hidden layers depending on the depth of the network requirements. The intermediate layers are linear layers with respective activation functions. The final layer returns to the same size as the original input layer, thus reconstructing the original data. At theDenoising Broadcast NetworksThe final layer consists of two separate outputs, each dedicated to the mean and variance of the predicted probability density, respectively.

5.2. Calculation of the loss function

The objective of the network model is to optimize the following loss function:

A reduced form of this loss function was proposed inSohl-Dickstein et al.[1] which formulates the loss in terms of a linear combination of KL divergence between two Gaussian distributions and a set of entropies. This simplifies the calculation and facilitates the implementation of the loss function. The loss function then becomes:

Further simplification and improvement were proposed byLo et al.[3] in the loss function in which the mean parameterization is used as described in the previous section for the direct process. Therefore, the loss function becomes:

5.3. Results

The results of the direct process that adds Gaussian noise following aMarkov chaincan be seen in the following figure. The total number of time steps was 100, whereas this figure shows 10 samples from the generated sequence set.

The results for the reverse diffusion process can be seen in the figure below. The quality of the final result depends on the adjustment of the hyperparameters and the number of training epochs.

6. conclusions

In this article, we discuss the fundamentals of diffusion models and their implementation. Although broadcast models are computationally more expensive than other deep network architectures, they work much better in certain applications. For example, in recent applications for text and image synthesis tasks, diffusion models have outperformed other architectures [4]. More implementation details and code can be found in the following github repository:https://github.com/azad-academy/denoising-difusión-modelo.git

Subscribe and follow for more updates:azad-wolf.medium.com/

References

[1]Sohl-Dickstein, J., Weiss, E.A., Maheswaranathan, N. and Ganguli, S. (2015). Unsupervised deep learning using nonequilibrium thermodynamics. arXiv preprint arXiv:1503.03585.

[2] Max Welling e Yee Whye Teh."Bayesian learning via langevin gradient stochastic dynamics".CIML 2011.

[3]Ho, J., Jain, A. and Abbeel, P. (2020).Noiseless diffusion probabilistic models. prepress arXiv arXiv:2006.11239.

[4]Prafulla Dhariwal,alex nichol,Diffusion models outperform GANs in image synthesis, arXiv: 2105.05233