Shouldn't an autoencoder with #(neurons in hidden layer) = #(neurons in input layer) be “perfect”?

Question

I'm exploring autoencoders for the first time. I'm using the Matlab neural networks toolbox. I have created a synthetic dataset consisting of points in 2D space plus some noise. My idea was to visualize the features of the hidden layer of the autoencoder to get the feeling about how it works. After training the autoencoder, I apply it to the same training data set to check the output.

If I consider one neuron in the hidden layer, then I get the output data all clustered on a straight line. Which makes sense to me, as with one degree of freedom (plus bias), there is not much else the autoencoder can do other than to find the line that minimizes the distance to the data points.

However, if I have two neurons in the hidden layer, I would expect the autoencoder to be perfect, since we have two degrees of freedom to match all the points. Hence, with enough epochs, it should be able to reduce the error to almost zero, and reproduce the input data perfectly. To my surprise, this doesn't happen: the training quickly stalls and we see some imperfections when trying to reproduce the input data.

So where's the catch ? Shouldn't an autoencoder be able to trivially replicate the input to the output when it doesn't have to compress the input information ?

PS: in the picture below, "# hidden layers" actually means "# of neurons in the hidden layers"

Answer 1

I think a part of your problem could be related to the activation function - you're probably using tanh or relu for the hidden layer. If the middle layer did not have an activation function, the network would learn to simply pass the information forward - ideally, there would be a weight of 1 between every input and a corresponding hidden layer neuron and the other weights would be 0, completely replicating the information. But since you have activation functions, this replication wouldn't work, the activation would distort the result...

The activation function, which breaks the linearity of the problem is what makes NN so powerful, but it also causes these problems.. they are also the cause that NN have multople local minimum, as Lucas said, so the NN can get stuck in a "good enough" solution.

In short: remove the activaction function (make it linear) and it should work...

Answer 2

Yes, you can have a perfect autoencoder if the number of hidden units is the same as the input. In this case, you could simply make the hidden neuron h_k consider the input i_k and ignore all other inputs (i.e., simply pass forward input i_k), and with linear activation functions you would get a perfect autoencoder.

It seems to me, though, that your problem is more related to the training process. As the weights are randomly initialized, it looks like the training may be getting stuck in a local minimum. I think we would need to know more details about the training parameters you are using.

Update: I'm thinking now that this looks like a convex problem, so there should be no local minimum. What optimizer and training parameters are you using?