I'm exploring autoencoders for the first time. I'm using the Matlab neural networks toolbox. I have created a synthetic dataset consisting of points in 2D space plus some noise. My idea was to visualize the features of the hidden layer of the autoencoder to get the feeling about how it works. After training the autoencoder, I apply it to the same training data set to check the output.
If I consider one neuron in the hidden layer, then I get the output data all clustered on a straight line. Which makes sense to me, as with one degree of freedom (plus bias), there is not much else the autoencoder can do other than to find the line that minimizes the distance to the data points.
However, if I have two neurons in the hidden layer, I would expect the autoencoder to be perfect, since we have two degrees of freedom to match all the points. Hence, with enough epochs, it should be able to reduce the error to almost zero, and reproduce the input data perfectly. To my surprise, this doesn't happen: the training quickly stalls and we see some imperfections when trying to reproduce the input data.
So where's the catch ? Shouldn't an autoencoder be able to trivially replicate the input to the output when it doesn't have to compress the input information ?
PS: in the picture below, "# hidden layers" actually means "# of neurons in the hidden layers"