18

With the help of better explanations provided in Z-Forcing: Training Stochastic Recurrent Networks: When posterior is not collapsed, $z_d$ (d-th dimension of latent variable $z$) is sampled from $q_{\phi}(z_d|x)=\mathcal{N}(\mu_d, \sigma^2_d)$, where $\mu_d$ and $\sigma_d$ are stable functions of input $x$. In other words, encoder distills useful ...


16

There is no specific constraint on the symmetry of an autoencoder. At the beginning, people tended to enforce such symmetry to the maximum: not only the layers were symmetrical, but also the weights of the layers in the encoder and decoder where shared. This is not a requirement, but it allows to use certain loss functions (i.e. RBM score matching) and can ...


13

Clustering is difficult to do in high dimensions because the distance between most pairs of points is similar. Using an autoencoder lets you re-represent high dimensional points in a lower-dimensional space. It doesn't do clustering per se - but it is a useful preprocessing step for a secondary clustering step. You would map each input vector $x_i$ to a ...


12

Your weights have diverged during training, and the network as a result is essentially broken. As it consists of ReLUs, I expect the huge loss in the first epoch caused an update which has zeroed out most of the ReLU activations. This is known as the dying ReLU problem, although the issue here is not necessarily the choice of ReLU, you will probably get ...


11

It is certainly correct in the sense that it is a legitimate neural network. The dropout layer introduces noise that is not injected during the test period. The goal is to combat overfitting so that the error in your test set will be lower due to better generalization. Applying the dropout layer on top of the input layer however throws away a lot of ...


11

what does an auto-encoder do? The simplest auto-encoder takes a high dimensional image (say, 100K pixels) down to a low-dimensional representation (say, a vector of length 10) and then uses only those 10 features to try to reconstruct the original image. You can imagine an analogy with humans: I look at someone, describe them ("tall, dark-haired, ...&...


10

Yes, it makes sense to use CNNs with autoencoders or other unsupervised methods. Indeed, different ways of combining CNNs with unsupervised training have been tried for EEG data, including using (convolutional and/or stacked) autoencoders. Examples: Deep Feature Learning for EEG Recordings uses convolutional autoencoders with custom constraints to improve ...


10

In general an autoencoder should perform well, when it comes to detect fraud examples. Fraud examples should have in theory a much higher reconstruction error. When it comes to train the autoencoder on binary data, I agree with you that it can be quite challenging. I suggest to take a look at this blog: https://blog.evjang.com/2016/11/tutorial-categorical-...


10

Yes. Two changes are required to convert an AE to VAE, which shed light on their differences too. Note that if an already-trained AE is converted to VAE, it requires re-training, because of the following changes in the structure and loss function. Network of AE can be represented as $$x \overbrace{\rightarrow .. \rightarrow y \overset{f}{\rightarrow}}^{\...


10

It probably won't. The whole point of the training was to encode cat images and thus the network has tried to learn what information is the most necessary to keep to ensure a low reconstruction error (i.e. what separates one cat from another) and what information can it throw away (i.e. what characteristics appear in all cat images and can be discarded). ...


10

An easy way to think about autoencoders is: how well a prticlar pice of infrmaton can be reconstrcted frm its reducd or otherwse comprssed reprsentaton. If you made it this far it means that you sucessfully reconstructed the previous sentence by using only 92 of its original 103 characters. More specifically, autoencoders are neural networks that are trained ...


9

According to section 2.3 of this paper: Auto-encoders are special cases of encoder-decoder models in which the input and output are the same. The same section of the paper describes encoder-decoder as follows: Encoder-Decoder models are a family of models which learn to map data-points from an input domain to an output domain via a two-stage network: ...


8

Your question is definitely in place, however I found that any question in the format of "should I do X or Y in deep learning ?" has only one answer. Try them both Deep learning is a very empirical field, and if a non-symmetric auto-encoder works for your domain, then use it (and publish a paper)


7

Yes to both of your questions. Your autoencoder can overfit and this will cause your bottleneck to store useless information (besides any useful information it already stores). Some ways to prevent this is: Find a larger dataset, or augment the current. Add noise to the input (see de-noising autoencoders). Regularization (e.g. early stopping, sparsity ...


6

Both types of networks try to reconstruct the input after feeding it through some kind of compression / decompression mechanism. For outlier detection the reconstruction error between input and output is measured - outliers are expected to have a higher reconstruction error. The main difference seems to be the way how the input is compressed: Plain ...


5

I have used stacked auto-encoders to reduce our 40 features step by step to 5 features and then output back to 40 features (some of my features were all zeros/ non deviating features). Training this on original (assumed to have no outliers) gives you a network which has learnt an abstract representation of the 40 features with 5 features. When outliers show ...


5

A possible approach would be a denoising autoencoder. It is like a normal autoencoder but instead of training it using the same input and output, you inject noise on the input while keeping the expected output clean. Hence, the autoencoder learns to remove it. This kind of autoencoders are also described in the blog post you linked to. In your case, you ...


5

I don't think that your assumption $w' = w^T$ holds. Or rather is not necessary, and if it is done, it is not in order to somehow automatically reverse the calculation to create the hidden layer features. It is not possible to reverse the compression in general, going from n to smaller m, directly in this way. If that was the goal, then you would want a form ...


5

The "autoassociative" paper is from 1992. The field had not settled on the term "autoencoder" for this concept.


5

I don't like the reduce_sum version of the kl-loss because it depends on the size of your latent vector. My advise is to use the mean instead. Moreover it is a notorious fact that training a VAE with the kl loss is difficult. You may need to progressively increase the contribution of the kl loss in your total loss. Add a weight w_kl that will control the ...


5

Yes, it is possible. You need to: Convert the bottleneck into a stochastic bottleneck. In VAE's the bottleneck are not the values deterministically generated by the encoder. Instead, the encoder generates the parameters defining some random variables. These random vars normally follow independent Gaussian distributions, and the encoder generates a vector ...


5

To have a common mental image of AE and VAE please take a look at this answer first. Lets go through this "why not?" thought process step by step: Why not deterministic? lets directly encode the latent vector $z$ inside a layer of neural network. But this way, $z$ would be deterministic, meaning a fix input $x$ always produces a fix latent vector $z$, thus,...


5

Auto Encoders are a special case of encoder-decoder models. In the case of auto encoders, the input and the output domains are the same ( typically ). The Wikipedia page for Autoencoder, mentions, The simplest way to perform the copying task perfectly would be to duplicate the signal. Instead, autoencoders are typically forced to reconstruct the input ...


5

Hey so the Keras implementation of Cosine Similarity is called as Cosine Proximity. It just has one small change, that being cosine proximity = -1*(Cosine Similarity) of the two vectors. This is done to keep in line with loss functions being minimized in Gradient Descent. To elaborate, Higher the angle between x_pred and x_true. lower is the cosine value. ...


4

Yes, but no-one can tell if they will work well for your problem, so just try it and see. Don't give up if it does not work at first, because training neural networks requires some practice; there are lots of parameters, and not every configuration will work well. Even the optimization algorithm is a hyperparameter.


4

I've put together some example tensorflow code to help explain (the full, working code is in this gist). This code implements the capsule network from the first part of section 2 in the paper you linked: N_REC_UNITS = 10 N_GEN_UNITS = 20 N_CAPSULES = 30 # input placeholders img_input_flat = tf.placeholder(tf.float32, shape=(None, 784)) d_xy = tf....


4

Autoencoders are a neural network solution to the problem of dimensionality reduction. The point of dimensionality reduction is to find a lower-dimensional representation of your data. For example, if your data includes people's height, weight, trouser leg measurement and shoe size, we'd expect there to be some underlying size dimension which would capture ...


4

Yes, you can use a convolutional network in an autoencoder setup. There is nothing strange with it. People have problems figuring out deconvolution layers, though. Here you can find an example of a convolutional autoencoder using Keras framework: https://blog.keras.io/building-autoencoders-in-keras.html


4

Does that mean that my model (or indeed my approach of using an AE) is ineffective Well, it depends. Auto Encoder are a quite broad field, there are many hyperparameters to tune, width, depth, loss function, optimizer, epochs. How should I proceed? From my gut feeling I would say that you don't have enough data to train the AE properly. Keep in Mind, ...


4

A working example of a Variational Autoencoder for Text Generation in Keras can be found here. Cross-entropy loss, aka log loss, measures the performance of a model whose output is a probability value between 0 and 1 for classification. Cross-entropy loss goes up as the predicted probability diverges from the actual label. In the case of character-by-...


Only top voted, non community-wiki answers of a minimum length are eligible