I am trying to train a neural network to find a mapping(embedding) to a lower dimensional space. I would like for my dataset, once mapped to the lower dimensional space, to appear gaussian-like distributed.
Does anyone have any idea what form of regularization I could use. I have been reading-up on KL-divergence but that seems to require a parametrized gaussian (rather than samples). Any help would be tremendously appreciated!
One easy approach is to use a modified VAE loss with a stochastic encoder. In other words, let $z=E(x)\in\mathbb{R}^n$ and $y = D(z)$ be your pseudo-decoder (not a decoder, since you said your output is not meant to perform reconstruction; in fact it's not strictly necessary at all). Then, as in VAEs, define the probabilistic encoder $E(x)$ via $$ \mu(x) = f_{\theta_1}(x), \;\; \Sigma(x) = g_{\theta_2}(x), \;\; z\sim \mathcal{N}(\mu,\Sigma) $$ where $\Sigma$ is a diagonal covariance matrix. Use the reparametrization trick for backprop. Commonly this is written $z\sim q_\theta(z|x)$.
Now, let your current loss (not necessarily reconstruction) be written $\mathcal{L}(x,z,y)$. Our "pseudo-VAE" regularized loss for one data point is then just: $$ \mathfrak{L}(x)=\mathcal{L}(x,z,y) + \beta\, \mathcal{L}_\text{KL}(x,z) $$ where $\beta\in\mathbb{R}^+$ and the KL loss is written as $$ \mathcal{L}_\text{KL}(x,z) = \frac{1}{2}\left[ -\log|\Sigma(x)| - n + \text{tr}(\Sigma) + \mu^T\mu \right] $$ assuming you want to be $\mathcal{N}(0,1)$ distributed.
Alternatively, you can treat each minibatch as approximating the marginal distribution, and then do moment matching. Concretely, let $X=(x_1,\ldots,x_n)$ be a minibatch of inputs, and $E(X)=Z=(z_1,\ldots,z_n)$ be the corresponding batch of encodings (embeddings). Suppose you want to make it "look like" $\mathcal{N}(\mu,\Sigma)$. Then let $$ \widehat{\mu}(Z)=\frac{1}{n}\sum_i z_i \;\;\;\;\;\&\;\;\;\;\; \widehat{\Sigma}(Z) =\frac{1}{n-1}\sum_i (z_i - \widehat{\mu}(Z))(z_i - \widehat{\mu}(Z))^T $$ so that our moment matching penalty (which you can add to your current loss function) is just $$ \mathcal{L}_\text{M}(X) = \alpha |\mu - \widehat{\mu}|^2 + \gamma || \Sigma - \widehat{\Sigma}(Z) ||_F^2 $$
Another, slightly more complex approach, is to use an adversarial method, as in adversarial auto-encoders, adversarially learned inference (ALI), etc... Basically, you train another network $C(z)$, which you train like the discriminator of a GAN (e.g., LS-GAN, WGAN-GP), but your generator is just $E(x)$. You train your critic $C$ to differentiate $ z\sim\mathcal{N}(0,I_n) $ and $z=E(x)$, and maximize its loss when training the "generator". This is equivalent to a conditional GAN, where the noise input is fixed (a Dirac Delta) or non-existent, the output is just the embedding, and the true samples are simply normally distributed random vectors.
One approach would be t-distributed Stochastic Neighbor Embedding (t-sne) to learn the low-dimensional embedding.
The perplexity hyperparameter would have been tuned to get the outcome you desire. And since the method is probabilistic, it might have to be run multiple times to yield the most useful embedding.
The data in the low-dimensional embedding space could be evaluated with K-L divergence to see if it is normal enough.
