29 votes
Accepted

What is "posterior collapse" phenomenon?

With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks: When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\...
Esmailian's user avatar
  • 9,312
20 votes
Accepted

Why are autoencoders for dimension reduction symmetrical?

There is no specific constraint on the symmetry of an autoencoder. At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights ...
noe's user avatar
  • 26.6k
14 votes
Accepted

Why my training and validation loss is not changing?

Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out ...
Neil Slater's user avatar
  • 28.9k
14 votes
Accepted

How can autoencoders be used for clustering?

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-...
tom's user avatar
  • 2,248
13 votes
Accepted

What is an autoencoder?

what does an auto-encoder do? The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only ...
cag51's user avatar
  • 447
11 votes

Validation loss is lower than the training loss

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that ...
Jan van der Vegt's user avatar
10 votes
Accepted

Does it make sense to train a CNN as an autoencoder?

Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (...
robintibor's user avatar
10 votes
Accepted

Using an autoencoder for anomaly detection on categorical data

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder ...
Andreas Look's user avatar
10 votes
Accepted

Transform an Autoencoder to a Variational Autoencoder?

Yes. Two changes are required to convert an AE to VAE, which shed light on their differences too. Note that if an already-trained AE is converted to VAE, it requires re-training, because of the ...
Esmailian's user avatar
  • 9,312
10 votes

What is the difference between an autoencoder and an encoder-decoder?

According to section 2.3 of this paper: Auto-encoders are special cases of encoder-decoder models in which the input and output are the same. The same section of the paper describes encoder-...
Mukul's user avatar
  • 201
10 votes
Accepted

How does an encoder-decoder network work?

It probably won't. The whole point of the training was to encode cat images and thus the network has tried to learn what information is the most necessary to keep to ensure a low reconstruction error (...
TmBrdy's user avatar
  • 356
10 votes

What is an autoencoder?

An easy way to think about autoencoders is: how well a prticlar pice of infrmaton can be reconstrcted frm its reducd or otherwse comprssed reprsentaton. If you made it this far it means that you ...
hH1sG0n3's user avatar
  • 2,038
9 votes

Why are autoencoders for dimension reduction symmetrical?

Your question is definitely in place, however I found that any question in the format of "should I do X or Y in deep learning ?" has only one answer. Try them both Deep learning is a very empirical ...
Ankit Suri's user avatar
9 votes
Accepted

What is the difference between an autoencoder and an encoder-decoder?

Auto Encoders are a special case of encoder-decoder models. In the case of auto encoders, the input and the output domains are the same ( typically ). The Wikipedia page for Autoencoder, mentions, ...
Shubham Panchal's user avatar
8 votes
Accepted

Difference: Replicator Neural Network vs. Autoencoder

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output ...
stmax's user avatar
  • 1,637
7 votes
Accepted

How can I prove bottleneck layer of my CNN auto encoder contain useful information?

Yes to both of your questions. Your autoencoder can overfit and this will cause your bottleneck to store useless information (besides any useful information it already stores). Some ways to prevent ...
cat91's user avatar
  • 413
6 votes

Autoencoder: using cosine distance as loss function

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being ...
Nasheed Yasin's user avatar
6 votes

Variational Autoencoders VS Transformers

Variational AutoEncoder VAE is an autoencoder whose encodings distribution is regularised during the training in order to ensure that its latent space has good properties allowing us to generate some ...
Pluviophile's user avatar
  • 3,848
5 votes
Accepted

Autoencoder for cleaning outliers in a surface

A possible approach would be a denoising autoencoder. It is like a normal autoencoder but instead of training it using the same input and output, you inject noise on the input while keeping the ...
noe's user avatar
  • 26.6k
5 votes
Accepted

Unsupervised feature reduction for anomaly detection with autoencoders

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this ...
rishiehari's user avatar
5 votes
Accepted

what's the difference between autoencoder and autoassociative neural networks?

The "autoassociative" paper is from 1992. The field had not settled on the term "autoencoder" for this concept.
Jack Parsons's user avatar
5 votes

Latent loss in variational autoencoder drowns generative loss

I don't like the reduce_sum version of the kl-loss because it depends on the size of your latent vector. My advise is to use the mean instead. Moreover it is a ...
Adrien D's user avatar
  • 1,113
5 votes

Transform an Autoencoder to a Variational Autoencoder?

Yes, it is possible. You need to: Convert the bottleneck into a stochastic bottleneck. In VAE's the bottleneck are not the values deterministically generated by the encoder. Instead, the encoder ...
noe's user avatar
  • 26.6k
5 votes

Variational auto-encoders (VAE): why the random sample?

To have a common mental image of AE and VAE please take a look at this answer first. Lets go through this "why not?" thought process step by step: Why not deterministic? lets directly encode the ...
Esmailian's user avatar
  • 9,312
5 votes
Accepted

How does variational autoencoders actually work in comparison to GAN?

VAEs were a hot topic some years ago. They were known to generate somewhat blurry images and sometimes suffered from posterior collapse (the decoder part ignores the bottleneck). These problems ...
noe's user avatar
  • 26.6k
4 votes
Accepted

do autoencoders work well for non images?

Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are ...
Emre's user avatar
  • 10.5k
4 votes

Does it make sense to train a CNN as an autoencoder?

Yes, you can use a convolutional network in an autoencoder setup. There is nothing strange with it. People have problems figuring out deconvolution layers, though. Here you can find an example of a ...
noe's user avatar
  • 26.6k
4 votes
Accepted

Does anyone could help me to understand what is the autoencoders?

Autoencoders are a neural network solution to the problem of dimensionality reduction. The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, ...
R Hill's user avatar
  • 1,105
4 votes
Accepted

Tips and tricks for designing time-series variational autoencoders

I can speak from a more theoretical point of view, but honestly I haven't had much success with VAEs. 1) How deep should my encoder and decoder network be? Are there any good guidelines? That ...
StatsSorceress's user avatar
4 votes

Why are autoencoders for dimension reduction symmetrical?

I did some extensive experimenting to address the asked question. My experiments indicated that the encoding path (left leg of the NN) should have less, but wider layers. I usually take half as many ...
JamesLi's user avatar
  • 41

Only top scored, non community-wiki answers of a minimum length are eligible