27
votes
Accepted
What is "posterior collapse" phenomenon?
With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks:
When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\...
- 9,147
20
votes
Accepted
Why are autoencoders for dimension reduction symmetrical?
There is no specific constraint on the symmetry of an autoencoder.
At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights ...
- 22.3k
14
votes
Accepted
Why my training and validation loss is not changing?
Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out ...
- 28.4k
14
votes
Accepted
How can autoencoders be used for clustering?
Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-...
- 2,238
13
votes
Accepted
What is an autoencoder?
what does an auto-encoder do?
The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only ...
- 447
11
votes
Validation loss is lower than the training loss
It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
- 9,328
10
votes
Accepted
Does it make sense to train a CNN as an autoencoder?
Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (...
- 326
10
votes
Accepted
Using an autoencoder for anomaly detection on categorical data
In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error.
When it comes to train the autoencoder ...
- 921
10
votes
Accepted
Transform an Autoencoder to a Variational Autoencoder?
Yes. Two changes are required to convert an AE to VAE, which shed light on their differences too. Note that if an already-trained AE is converted to VAE, it requires re-training, because of the ...
- 9,147
10
votes
What is the difference between an autoencoder and an encoder-decoder?
According to section 2.3 of this paper:
Auto-encoders are special cases of encoder-decoder models in which the input and output are the same.
The same section of the paper describes encoder-...
- 201
10
votes
Accepted
How does an encoder-decoder network work?
It probably won't. The whole point of the training was to encode cat images and thus the network has tried to learn what information is the most necessary to keep to ensure a low reconstruction error (...
- 356
10
votes
What is an autoencoder?
An easy way to think about autoencoders is: how well a prticlar pice of infrmaton can be reconstrcted frm its reducd or otherwse comprssed reprsentaton. If you made it this far it means that you ...
- 1,978
9
votes
Why are autoencoders for dimension reduction symmetrical?
Your question is definitely in place, however I found that any question in the format of "should I do X or Y in deep learning ?" has only one answer.
Try them both
Deep learning is a very empirical ...
- 116
9
votes
Accepted
What is the difference between an autoencoder and an encoder-decoder?
Auto Encoders are a special case of encoder-decoder models. In the case of auto encoders, the input and the output domains are the same ( typically ). The Wikipedia page for Autoencoder, mentions,
...
- 2,140
8
votes
Accepted
Difference: Replicator Neural Network vs. Autoencoder
Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
- 1,637
7
votes
Accepted
How can I prove bottleneck layer of my CNN auto encoder contain useful information?
Yes to both of your questions. Your autoencoder can overfit and this will cause your bottleneck to store useless information (besides any useful information it already stores).
Some ways to prevent ...
- 413
6
votes
Autoencoder: using cosine distance as loss function
Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being ...
- 161
6
votes
Variational Autoencoders VS Transformers
Variational AutoEncoder
VAE is an autoencoder whose encodings distribution is regularised during the training in order to ensure that its latent space has good properties allowing us to generate some ...
- 3,550
5
votes
Accepted
Unsupervised feature reduction for anomaly detection with autoencoders
I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
- 166
5
votes
Accepted
Autoencoder for cleaning outliers in a surface
A possible approach would be a denoising autoencoder. It is like a normal autoencoder but instead of training it using the same input and output, you inject noise on the input while keeping the ...
- 22.3k
5
votes
Why is Reconstruction in Autoencoders Using the Same Activation Function as Forward Activation, and not the Inverse?
I don't think that your assumption $w' = w^T$ holds. Or rather is not necessary, and if it is done, it is not in order to somehow automatically reverse the calculation to create the hidden layer ...
- 28.4k
5
votes
Accepted
what's the difference between autoencoder and autoassociative neural networks?
The "autoassociative" paper is from 1992. The field had not settled on the term "autoencoder" for this concept.
- 116
5
votes
Latent loss in variational autoencoder drowns generative loss
I don't like the reduce_sum version of the kl-loss because it depends on the size of your latent vector. My advise is to use the mean instead.
Moreover it is a ...
- 1,083
5
votes
Transform an Autoencoder to a Variational Autoencoder?
Yes, it is possible. You need to:
Convert the bottleneck into a stochastic bottleneck. In VAE's the bottleneck are not the values deterministically generated by the encoder. Instead, the encoder ...
- 22.3k
5
votes
Variational auto-encoders (VAE): why the random sample?
To have a common mental image of AE and VAE please take a look at this answer first.
Lets go through this "why not?" thought process step by step:
Why not deterministic? lets directly encode the ...
- 9,147
5
votes
Accepted
How does variational autoencoders actually work in comparison to GAN?
VAEs were a hot topic some years ago. They were known to generate somewhat blurry images and sometimes suffered from posterior collapse (the decoder part ignores the bottleneck). These problems ...
- 22.3k
4
votes
Accepted
do autoencoders work well for non images?
Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are ...
- 10.5k
4
votes
Transforming AutoEncoders
I've put together some example tensorflow code to help explain (the full, working code is in this gist). This code implements the capsule network from the first part of section 2 in the paper you ...
- 568
4
votes
Accepted
Does anyone could help me to understand what is the autoencoders?
Autoencoders are a neural network solution to the problem of dimensionality reduction.
The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, ...
- 1,105
4
votes
Does it make sense to train a CNN as an autoencoder?
Yes, you can use a convolutional network in an autoencoder setup. There is nothing strange with it. People have problems figuring out deconvolution layers, though.
Here you can find an example of a ...
- 22.3k
Only top scored, non community-wiki answers of a minimum length are eligible