I am new to machine learning and experimenting with neural networks. I would like to build a network that takes as an input a series of 5 images and predicts the next image. My data set is completely artificial, just for my experimentation. As an illustration, here are a couple of examples of input and expected output:
The images of the data points and of the targets are from the same source: the target image of a data point appears in other data points, and vice-versa.
What I have done:
For now I have built a perceptron with one hidden layer and the output layer gives the pixels of the prediction. The two layers are dense and made of sigmoid neurons and I used the mean squared error as the objective. As the images are fairly simple and do not vary much, this does the job well: with 200-300 examples and 50 hidden units, I get a good error value (0.06) and good predictions on test data. The network is trained with gradient descent (with learning rate scaling). Here are the kinds of learning curves I get, and the error evolution with number of epochs:
What I am trying to do:
This is all good, but now I would like to reduce the dimensionality of the data set so that it would scale to bigger images and more examples. So I applied PCA. However I did not apply it on the list of data points but on the list of images, for two reasons:
- On the data set as a whole, the convariance matrix would be 24000x24000, which does not fit in the memory of my laptop;
- By doing on the images, I can also compress the targets, since they are made of the same images.
As the images look all similar, I managed to reduce their size from 4800 (40x40x3) to 36 while loosing only 1e-6 of the variance.
What does not work:
When I feed my reduced data set and its reduced targets to the network, the gradient descent converges very fast to a high error (around 50 !). You can see the equivalent plots as the ones above:
I had not imagined that a learning curve could start at a high value, and then go down and back up... And what are the usual causes of gradient descent stopping so fast? Could it be linked to parameter initialization (I use GlorotUniform, the default of the lasagne library).
Then I noticed that if I feed the reduced data but the original (uncompressed) targets, I get back the initial performance. So it seems that applying PCA on the target images was not a good idea. Why is that? After all, I just multiplied the inputs and the targets by the same matrix, so the training input and target are still linked in a manner that the neural network should be able to figure out, no? What am I missing?
Even if I introduce and extra layer of 4800 units so that there is the same total number of sigmoid neurons, I get the same results. To sum up, I have tried:
- 24000 pixels => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
- 180 "pixels" => 50 sigmoids => 36 sigmoids (= 36 "pixels")
- 180 "pixels" => 50 sigmoids => 4800 sigmoids (= 4800 pixels)
- 180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 sigmoids (= 36 "pixels")
- 180 "pixels" => 50 sigmoids => 4800 sigmoids => 36 linear (= 36 "pixels")
(1) and (3) work fine; but not (2), (4) and (5), and I don't understand why. In particular, since (3) works, (5) should be able to find the same parameters as (3) and the eigen vectors in the last linear layer. Is that not possible for a neural network?