Why does applying PCA on targets causes underfitting?

Question

The goal:

I am new to machine learning and experimenting with neural networks. I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:

The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.

What I have done:

For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:

What I am trying to do:

This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of images, for two reasons:

1. On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;
2. By doing on the images, I can also compress the targets, since they are made of the same images.

As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance.

What does not work:

When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:

I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).

Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?

Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:

1. 24000 pixels => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
2. 180 "pixels" => 50 sigmoids => 36 sigmoids (= 36 "pixels")
3. 180 "pixels" => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
4. 180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 sigmoids (= 36 "pixels")
5. 180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 linear (= 36 "pixels")

(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?

Answer 1

First, thanks for the edits to your original question since we now know that you are applying the same transformation to all of your data.

Q: Why do perceptrons perform so much better than generalized linear models for some problems? A: Because they are inherently nonlinear models, with a great deal of flexibility. The drawback is that the additional knobs require more data to correctly tune.

Big picture:

Less data can cause high-bias. High-bias can be overcome by more data. You have reduced your data from a 4800 feature dataset to a 38 feature dataset, so should expect to see increased bias. Neural networks require more data than models without hidden layers.

Linearity vs. Nonlinearity

Your artificial neural network (perceptron) is an inherently nonlinear model, yet you are deciding to remove features from your dataset with a linear model (PCA). The presence of a single hidden layer explicitly creates second order terms in your data, and there are two additional nonlinear transformations (input==>hidden and hidden==>prediction), which add sigmoidal nonlinearity at each step.

The fact that the training data and the target data are both multiplied by the same matrix to achieve the reduced dimensionality really just means that your perceptron needs to learn the matrix if it wants to reconstruct nonlinear aspects of the original data. This requires more data or for you to reduce less of the dimensionality.

An Experiment:

I suggest trying an experiment where you perform second order polynomial feature extraction on the full dataset and then performing PCA on that enhanced dataset. See how many features you end up with while retaining 99% of the variance of the enhanced dataset. If it is larger than the dimensionality of your initial dataset then stick with the un-enhanced and un-reduced dataset. If it is betweeen the dimensionality of the original data and 38 then try training the perceptron with that data.

A better idea:

Rather than using the (linear) variance to determine the feature reduction of your PCA projection, try training and cross validating your model with different amounts of PCA dimensionality reduction. You will likely find that there is a sweet spot for a given set of images. For instance, an SVM on the MNIST digit data performs best when the 784 pixel features are reduced down to 50 linearly independent features using PCA. Though this is not obvious from analyzing the variance of the principal components.

Other options:

There are nonlinear dimensionality reduction techniques, such as isomap. You could investigate the use of nonlinear feature reduction since you are clearly loosing information with the linear PCA that you have applied.

You could also look into image specific feature extraction techniques to add some nonlinearity before reducing the dimensionality.

Hope this helps!

Answer 2

It is possible that most of the variance in the dataset exists between input images (or between input and output images). In that case, the most informative principal components serve to separate input examples or to separate the input from the output.

If only the less informative PCs describe the variance b/w outputs, it will be harder to distinguish between outputs than it was in the original feature space.

That said, PCA in images is rarely all that helpful. In this example I can see it being appealing, but it would probably much more meaningful and memory-friendly to learn a different lower dimensional feature representation. You could try a simple autoencoder, SIFT/SURF features, or haar-like features.

Answer 3

I've read your Post a few Times now. But I'm not absolutely sure if I understood your Experiments fully. But I have a guess what might be happening.

I think the error is hidden in the part where you achieve this astonishing 0.06 percent. You're not only generalizing to new data but you predict the future in a way. This should not work, but I think it does because your net is horribly overfitted. So to say your network is still very dumb. It just learned if you show it pictures a, b it will answer c. The second Experiment contains a twist, if I understood you correctly. I assume you still present complete images to the network, but they are built out of the coefficients of their eigenimages. Now it has to spit out coefficients to build the answer from eigenimages. But this would require some understanding of the process, which the network didn't aquire.

So I think there are a number of suggestions to improve your experiment:

The learning of complete inputimages is just an extremely high dimensional problem which requires huge ammount of training data. Consider calculating features from the image.

I'm not sure if your learning problem is a sound one. Please consider to separate your training data from your prediction data and take more conventional data like handwritten digits.