Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have ran a hyperparameter search for a denoising autoencoder and the results suggest I should make the sizes of my hidden layers as large as possible (within the range of values I allowed it to search over.) This makes sense intuitively (i think?), wider layers would allow it to more easily copy the output to the input. However, I thought this trivial result was something a denoising autoencoder was meant to prevent.
Can anyone offer any advice? Thanks.
Let's consider a simple hypothetical autoencoder with only a single hidden layer. The goal is to constrain the number of neurons in the hidden layer such that we can compress the information from the input into this smaller feature set without losing significant information.
This is formalized as a transformation from $\mathbb{R}^n$ to $\mathbb{R}^m$ where $m<n$.
The trivial solution with a denoising autoencoder or a simple autoencoder is to maximize the number of hidden layers such that information is not lost during the compression process. A denoising autoencoder does not protect you from that.
The compressed information at the output of the hidden layer should be rich and pertinent to the input distribution. However, with a simple autoencoder it happens that the neurons simply learn to copy the input values to the output without learning a meaningful representation in the compressed layer. This is where denoising autoencoders are useful. We can add noise to the inputs, such that when an output is produced we can be more confident that the hidden layer correctly encoded the input.
What you want to do when choosing the number of hidden layers for an autoencoder is to set yourself an acceptable error threshold for the reconstruction. Then find the minimum number of neurons needed to attain that threshold.
