What are the possible approaches to fixing Overfitting on a CNN?

Question

Currently I am trying to make a cnn that would allow for age detection on facial images. My dataset has the following shape where the images are grayscale.

(50000, 120, 120) - training 
(2983, 120, 120) - testing

And my model currently looks like the following - I've been testing/trying different methods.

    model = Sequential()
    model.add(Conv2D(64, kernel_size=3, use_bias=False,
                     input_shape=(size, size, 1)))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Conv2D(32, kernel_size=3, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(MaxPooling2D(pool_size=(2, 2)))
    model.add(Dropout(0.25))
    model.add(Flatten())

    model.add(Dense(128, use_bias=False))
    model.add(BatchNormalization())
    model.add(Activation("relu"))

    model.add(Dropout(0.5))
    model.add(Dense(10, activation='softmax'))


    #TODO: Add in a lower learning rate - 0.001
    adam = optimizers.adam(lr=0.01)
    model.compile(optimizer=adam, loss='categorical_crossentropy',
                  metrics=['accuracy'])
    model.fit(x_train, y_train, validation_data=(x_test, y_test),
              epochs=number_of_epochs, verbose=1)

After running my data on just 10 epochs I started to initially see decent values but at the end of the run my results were the following and it has me concerned that my model is definitely over fitting.

How many epochs: 10
Train on 50000 samples, validate on 2939 samples
Epoch 1/10
50000/50000 [==============================] - 144s 3ms/step - loss: 1.7640 - acc: 0.3625 - val_loss: 1.6128 - val_acc: 0.4100
Epoch 2/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5815 - acc: 0.4059 - val_loss: 1.5682 - val_acc: 0.4059
Epoch 3/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.5026 - acc: 0.4264 - val_loss: 1.6673 - val_acc: 0.4158
Epoch 4/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.3996 - acc: 0.4641 - val_loss: 1.5618 - val_acc: 0.4209
Epoch 5/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.2478 - acc: 0.5226 - val_loss: 1.6530 - val_acc: 0.4066
Epoch 6/10
50000/50000 [==============================] - 141s 3ms/step - loss: 1.0619 - acc: 0.5954 - val_loss: 1.6661 - val_acc: 0.4086
Epoch 7/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.8695 - acc: 0.6750 - val_loss: 1.7392 - val_acc: 0.3770
Epoch 8/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.7054 - acc: 0.7368 - val_loss: 1.8634 - val_acc: 0.3743
Epoch 9/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5876 - acc: 0.7848 - val_loss: 1.8785 - val_acc: 0.3767
Epoch 10/10
50000/50000 [==============================] - 141s 3ms/step - loss: 0.5012 - acc: 0.8194 - val_loss: 2.2673 - val_acc: 0.3981
Model Saved

I assume the issue might be related to the number of images I have for each output class, but other then that I am a bit stuck in moving forward. Is there something wrong in my understanding/implementation? Any advice or critique would be well appreciated this is more of a learning project for me.

Answer 1

Try to use dropout after your dense layers not after maxpooling layers. Whatever comes before dense layers can be considered as the inputs of a classification layer. So keep them otherwise it somehow means you are loosing appropriate information. You should also be aware that you should not use dropout after the last layer.

Also you can add another dense layer, two hidden dense layers, for classification. It seems your data is not easy to learn.

Answer 2

To deal with overfitting, you need to use regularization during the training:

1. Weight regularization - The first thing you have to do (practically always) is to use regularization on the weights of the model. L1 or L2 regularization update the general loss function by adding another term known as the regularization term. As a result thee values of weights decrease because it assumes that a neural network with smaller weights leads to simpler models. Therefore, it will also reduce overfitting. If you are not sure what you need, just use L2.

   Keras - Usage of regularizers

2. Dropout - Add dropout layers after dense layers (by the way, there are also advantages to using dropout after the convolution layers, it helps with occlusions). Just make sure not to use it at the final dense layer (the one with the same size as the number of classes).

3. Data Augmentation - The simplest way to reduce overfitting is to increase the size of the training data. Use data augmentation to potentially expend your training set to "infinity". Keras's data augmentation is really simple an easy to use:

   Keras Image Preprocessing

If you implement these 3 steps, you will see drastic improvements (probably even just after the first one).

Further corrections and improvements (nothing to do with overfitting):

• Your batch normalization layer should come after the non-linear activation, or more accurately, it needs to come before the next convolution layer.
• Add an additional dense layer or 2 (only if the results are not good enough).