-1
$\begingroup$

I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to search over.) This makes sense intuitively (i think?), wider layers would allow it to more easily copy the output to the input. However, I thought this trivial result was something a denoising autoencoder was meant to prevent.

Can anyone offer any advice? Thanks.

$\endgroup$
0
$\begingroup$

Let's consider a simple hypothetical autoencoder with only a single hidden layer. The goal is to constrain the number of neurons in the hidden layer such that we can compress the information from the input into this smaller feature set without losing significant information.

This is formalized as a transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ where $m<n$.

The trivial solution with a denoising autoencoder or a simple autoencoder is to maximize the number of hidden layers such that information is not lost during the compression process. A denoising autoencoder does not protect you from that.

The compressed information at the output of the hidden layer should be rich and pertinent to the input distribution. However, with a simple autoencoder it happens that the neurons simply learn to copy the input values to the output without learning a meaningful representation in the compressed layer. This is where denoising autoencoders are useful. We can add noise to the inputs, such that when an output is produced we can be more confident that the hidden layer correctly encoded the input.

What you want to do when choosing the number of hidden layers for an autoencoder is to set yourself an acceptable error threshold for the reconstruction. Then find the minimum number of neurons needed to attain that threshold.

$\endgroup$
  • $\begingroup$ thank you for the response. So, rather than doing a gridsearch for my denoising autoencoder to find the best number of neurons in the layers I should choose an error I am ok with and manually tune the autoencoder to find the least number of neurons that will give that result? $\endgroup$ – Andrew Oct 3 '18 at 9:39
  • $\begingroup$ Yes that's right. It is quite rare that you will find an example where compression will retain 100% of the original information. Thus choose a threshold and dont cross it $\endgroup$ – JahKnows Oct 4 '18 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.