1
$\begingroup$

When using an auto encoder to create non-linear dimensional reduced featires, is it more common to use the output of the network (the prediction of the input features) or to use the weights from the (or 1 of the if there are multiple) hidden layers? If the hidden layer is used, do you use the hidden layer activation as features or weights from the hidden layer to the output?

$\endgroup$
2
$\begingroup$

When you want to use Auto-Encoders (AEs) for dimensionality reduction, you usally add a bottleneck layer. This means, for example, you have 1234-dimensional data. You feed this into your AE, and - as it is an AE - you have an output of dimension 1234. However, you might have many layers in that network and one of them has significantly less dimensions. Lets say you have the topology 1234:1024:784:1024:1234. You train it like this, but you only use the weights from the 1234:1024:784 part.

When you get new input, you just feed it into this network. You can see it as a kind of preprocessing. For the later stages, this is a black box.

This is manly useful when you have a lot of unlabeled data. It is called Semi Supervised Learning (SSL).

| improve this answer | |
$\endgroup$
  • $\begingroup$ Are you using the activations from the 784 units as features? It seems from looking at various sources this is the case (but as will most things, there are different versions). I was also looking at the autoencoder for Keras and there they use the last hidden layer activations keras.io/layers/core/#autoencoder $\endgroup$ – B_Miner Jan 26 '16 at 14:23
  • $\begingroup$ I am not sure what your question is. Are you asking if the pre-activation of the last layer or post-activation values are taken? Then the answer is that it probably doesn't matter, as long as you do it consistantly. - And yes, you take those values as features. Just like you take the values of a first hidden layer as features for a second hidden layer. It is the same idea. $\endgroup$ – Martin Thoma Jan 26 '16 at 15:31
  • $\begingroup$ Your answer mentioned using the "weights" and I was looking for clarification if the nodes (activation) or the weights were used typically as features. The other part was if the last hidden layer (e.g. Keras) was used typically or the middle hidden layer as you mentioned. Thanks! $\endgroup$ – B_Miner Jan 26 '16 at 16:43
  • $\begingroup$ @B_Miner You use the activation, not the weights themselves, as features. The idea why AEs work is that they learn what is important in images. They learn a more compact representation of an image. The learning is done by gradually adjusting the weights. - In case you are looking for an introduction to neural networks, I can recommend neuralnetworksanddeeplearning.com $\endgroup$ – Martin Thoma Jan 26 '16 at 16:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.