Convolutional neural network overfitting. Dropout not helping

I am playing a little with convnets. Specifically, I am using the kaggle cats-vs-dogs dataset which consists on 25000 images labeled as either cat or dog (12500 each).

I've managed to achieve around 85% classification accuracy on my test set, however I set a goal of achieving 90% accuracy.

My main problem is overfitting. Somehow it always ends up happening (normally after epoch 8-10). The architecture of my network is loosely inspired by VGG-16, more specifically my images are resized to $$128x128x3$$, and then I run:

Convolution 1 128x128x32 (kernel size is 3, strides is 1)
Convolution 2 128x128x32 (kernel size is 3, strides is 1)
Max pool    1 64x64x32   (kernel size is 2, strides is 2)
Convolution 3 64x64x64   (kernel size is 3, strides is 1)
Convolution 4 64x64x64   (kernel size is 3, strides is 1)
Max pool    2 32x32x64   (kernel size is 2, strides is 2)
Convolution 5 16x16x128  (kernel size is 3, strides is 1)
Convolution 6 16x16x128  (kernel size is 3, strides is 1)
Max pool    3 8x8x128    (kernel size is 2, strides is 2)
Convolution 7 8x8x256    (kernel size is 3, strides is 1)
Max pool    4 4x4x256    (kernel size is 2, strides is 2)
Convolution 8 4x4x512    (kernel size is 3, strides is 1)
Fully connected layer 1024 (dropout 0.5)
Fully connected layer 1024 (dropout 0.5)


All the layers except the last one have relus as activation functions.

Note that I have tried different combinations of convolutions (I started with simpler convolutions).

Also, I have augmented the dataset by mirroring the images, so that in total I have 50000 images.

Also, I am normalizing the images using min max normalization, where X is the image

$$X = X - 0 / 255 - 0$$

The code is written in tensorflow and the batch sizes are 128.

The mini-batches of training data end up overfitting and having an accuracy of 100% while the validation data seems to stop learning at around 84-85%.

I have also tried to increase/decrease the dropout rate.

The optimizer being used is AdamOptimizer with a learning rate of 0.0001

At the moment I have been playing with this problem for the last 3 weeks and 85% seems to have set a barrier in front of me.

For the record, I know I could use transfer learning to achieve much higher results, but I am interesting on building this network as a self-learning experience.

Update:

I am running the SAME network with a different batch size, in this case I am using a much smaller batch size (16 instead of 128) so far I am achieving 87.5% accuracy (instead of 85%). That said, the network ends up overfitting anyway. Still I do not understand how a dropout of 50% of the units is not helping... obviously I am doing something wrong here. Any ideas?

Update 2:

Seems like the problem had to do with the batch size, as with a smaller size (16 instead of 128) I am achieving now 92.8% accuracy on my test set, with the smaller batch size the network still overfits (the mini batches end up with an accuracy of 100%) however, the loss (error) keeps decreasing and it is in general more stable. The cons are a MUCH slower running time, but it is totally worth the wait.

• Could you give more details around your assessment of over-fitting? For instance, does the validation accuracy drop at any point, alongside the divergence from training and validation results? How about the loss function? – Neil Slater Aug 23 '17 at 7:11
• Good question, so by overfitting I mean the fact that the mini-batches in train achieve accuracy of 100% and losses of 0.08 while validation does not seem to ever go under 0.35 and its accuracy remains now at 88%. As per the validation it does not seem to drop (at least not too much), seems to become flat, however how come the mini batch achieve such a low loss while validation is still far away from it? – Juan Antonio Gomez Moriano Aug 23 '17 at 22:04
• I don't know an answer for you, however this behaviour - large divergence between train and validation, but still OK-ish plateaued validation - is something I have seen before a few times. I almost hesitate to call it over-fitting because sometimes the test results are acceptable. – Neil Slater Aug 23 '17 at 22:11
• "Still I do not understand how a dropout of 50% of the units is not helping" I have seen people using much higher values of dropout with success. – Ricardo Cruz Aug 24 '17 at 23:58

Ok, so after a lot of experimentation I have managed to get some results/insights.

In the first place, everything being equal, smaller batches in the training set help a lot in order to increase the general performance of the network, as a negative side, the training process is muuuuuch slower.

Second point, data is important, nothing new here but as I learned while fighting this problem, more data always seems to help a bit.

Third point, dropout is useful in large networks with lots of data and lots of iterations, in my network I applied dropout on the final fully connected layers only, convolution layers did not get dropout applied.

Fourth point (and this is something I am learning over and over): neural networds take A LOT to train, even on good GPUs (I trained this network on floydhub, which uses quite expensive NVIDIA cards), so PATIENCE is key.

Final conclusion: Batch sizes are more important that one might think, apparently it is easier to hit a local minimum when batches are larger.

The code I wrote is available as a python notebook, I think it is decently documented

https://github.com/moriano/loco-learning/blob/master/cats-vs-dogs/cats-vs-dogs.ipynb

• Thanks for posting your findings. Quick question: I'm facing a similar issue and I saw this in the notebook you posted: NOTE USE EITHER mean centering or min-max, NOT BOTH. I'm currently dividing my input images by 255 inside my input_fn (Tensorflow Estimator API). Then, inside the model, I'm running that input through batch norm. Should I still only do one of those normalizations? See github.com/formigone/tf-imagenet/blob/master/models/… – rodrigo-silveira Aug 10 '18 at 1:52
• My understandingis that dividing by 255 is done only once to each image, and the reason is to keep all the values between 0 and 1 as that will provide numerical stability. – Juan Antonio Gomez Moriano Aug 10 '18 at 21:17
• Sure, I get that. But do you think it makes sense to also batch normalize those values in the range [0, 1]? – rodrigo-silveira Aug 10 '18 at 22:25
• That, I do not know, it has been a while since I have used batch normalization :) – Juan Antonio Gomez Moriano Aug 14 '18 at 22:54

I suggest you analyze the learning plots of your validation accuracy as Neil Slater suggested. Then, if the validation accuracy drops try to reduce the size of your network (seems too deep), add dropout to the CONV layers and BatchNormalization after each layer. It can help get rid of overfitting and increase the test accuracy.

• Thanks for the advice, will try it, I was however under the impression that CONV layers do not require dropout, on most of the papers I have read, dropout seems to be applied always to the end fully connected layers, not to the convolutins. – Juan Antonio Gomez Moriano Aug 23 '17 at 22:06

There are several possible solutions for your Problem.

1. Use Dropout in the earlier layers (convolutional layers) too.

2. Your network seems somehow quite big for such an "easy" task; try to reduce it. The big architectures are also trained on much bigger datasets.

If you want to keep your "big" architecture try:

1. Image augmentation in order to virtually increase your training data

2. Try adversarial training. It sometimes helps.

• "Your network seems somehow quite big for such an "easy" task; try to reduce it. The big architectures are also trained on much bigger datasets." I disagree, as I added more convolutions, the accuracy increased (initially i was achieving 68% with just two convolutions). Also, I am already augmentating my dataset, I operate with 50000 images. – Juan Antonio Gomez Moriano Aug 23 '17 at 22:07

One thing that hasn't been mentioned yet and that you can consider for the future: you can still increase your dropout at the fully connected layers.

I read a paper once that used 90% dropout rate. Although it had many many nodes (2048 if i recall correctly), I have tried this myself on layers with fewer nodes and it was very helpful in some cases.

I just looked up which paper it was. I can't recall which paper I just remembered but I found these that also had some success with 90% dropout rates.

Karpathy, A., Toderici, G., Shetty, S., Leung, T., Sukthankar, R., & Fei-Fei, L. (2014). Large-scale video classification with convolutional neural networks. In Proceedings of the IEEE conference on Computer Vision and Pattern Recognition (pp. 1725-1732).

Simonyan, K., & Zisserman, A. (2014). Two-stream convolutional networks for action recognition in videos. In Advances in neural information processing systems (pp. 568-576).

Varol, G., Laptev, I., & Schmid, C. (2017). Long-term temporal convolutions for action recognition. IEEE transactions on pattern analysis and machine intelligence.

I had this problem too. After dinking with it for hours, by chance I decided to shuffle the data before feeding it into the system and voila, it started working. It took me a bit to figure out that it was the shuffling that did the trick! Hope this saves somebody from frustration!