AE

VAE

reparameterization trick

Assuming the distribution of the output is a Gaussian distribution, the model only predicts the mean ($\mu$) and std ($\sigma$). We then sample the latent variable from this Gaussian distribution. The sample latent distribution parameters should match the true distribution, which is enforced using the KL divergence.

VAE loss

Evidence Lower Bound (ELBO)

We want replace $p_{\theta }(z|x)$ with $q_{\varphi }(z)$ since we dont have GT of $p_{\theta }(z|x)$

$$\begin{array}{rl}\log p_{\theta }(x)& =E_{(q_{\varphi })}[\log p_{\theta }(x)]\\ & =E_{q_{\varphi }(z)}[\log (\frac{p_{\theta }(x,z)}{p_{\theta }(z|x)})]\\ & =E_{q_{\varphi }(z)}[\log (\frac{p_{\theta }(x,z)}{p_{\varphi }(z)}\frac{p_{\varphi }(z)}{p_{\theta }(z|x)})]\\ & =E_{q_{\varphi }(z)}[\log (\frac{p_{\theta }(x,z)}{p_{\varphi }(z)})]+E_{q_{\varphi }(z)}[\log (\frac{p_{\varphi }(z)}{p_{\theta }(z|x)})]\\ & =E_{q_{\varphi }(z)}[\log (\frac{p_{\theta }(x,z)}{p_{\varphi }(z)})]+KL(q_{\varphi }(z)||p_{\theta }(z|x))\\ \log p_{\theta }(x)& =E_{q_{\varphi }(z)}[\log (\frac{p_{\theta }(x,z)}{p_{\varphi }(z)})]+KL(q_{\varphi }(z)||p_{\theta }(z|x))\\ \log p_{\theta }(x)& ={\mathcal{L}}_{ELBO}+KL(q_{\varphi }(z)||p_{\theta }(z|x))\\ {\mathcal{L}}_{ELBO}& =\log p_{\theta }(x)-KL(q_{\varphi }(z)||p_{\theta }(z|x))\end{array}$$

base on difference condition KL divergence can simplify to difference term

• Variational Inference
• Importance Sampling to ELBO
• Variational EM

refs

• kvfrans
• odie2630463
• bobondemon

VQVAE(d-vae)

Variational Autoencoder (Kingma & Welling, 2014)

quantise bottleneck

• random init centroids
• find the nearest centroids of each unquantise vector
  • if quantise have low usage then random init a new centroids
• calculate average center of unquantise vector
• Exponential Moving Average Update between new centroid and old centroid

loss

• VQ loss: The L2 error between the embedding space and the encoder outputs.
• Commitment loss: A measure to encourage the encoder output to stay close to the embedding space and to prevent it from fluctuating too frequently from one code vector to another.
• where $\text{sg}[.]$ is the stop_gradient operator.

$$L=\underset{\text{reconstruction loss}}{\underbrace{\Vert \mathbf{x}-D({\mathbf{e}}_k){\Vert }_2^2}}+\underset{\text{VQ loss}}{\underbrace{\Vert \text{sg}[E(\mathbf{x})]-{\mathbf{e}}_k{\Vert }_2^2}}+\underset{\text{commitment loss}}{\underbrace{\beta \Vert E(\mathbf{x})-\text{sg}[{\mathbf{e}}_k]{\Vert }_2^2}}$$

CVAE