Training accuracy is ~97% but validation accuracy is stuck at ~40%

Question

I am trying to classify images into 27 classes using a Conv2D network. The training accuracy rises through epochs as expected but the val_accuracy and val_loss values fluctuate severely and are not good enough.

I am using separate datasets for training and validation. The images are 256 x 256 in size and are binary threshold images.

There are 22127 images in the training set (~800 for each class) and 11346 images in the validation set (~400 for each class) so I believe there is no class imbalance.

Here is my architecture:

_________________________________________________________________
Layer (type)                 Output Shape              Param #   
=================================================================
conv2d_1 (Conv2D)            (None, 126, 126, 32)      320       
_________________________________________________________________
max_pooling2d_1 (MaxPooling2 (None, 63, 63, 32)        0         
_________________________________________________________________
conv2d_2 (Conv2D)            (None, 61, 61, 32)        9248      
_________________________________________________________________
max_pooling2d_2 (MaxPooling2 (None, 30, 30, 32)        0         
_________________________________________________________________
flatten_1 (Flatten)          (None, 28800)             0         
_________________________________________________________________
dense_1 (Dense)              (None, 128)               3686528   
_________________________________________________________________
dropout_1 (Dropout)          (None, 128)               0         
_________________________________________________________________
dense_2 (Dense)              (None, 96)                12384     
_________________________________________________________________
dropout_2 (Dropout)          (None, 96)                0         
_________________________________________________________________
dense_3 (Dense)              (None, 64)                6208      
_________________________________________________________________
dense_4 (Dense)              (None, 27)                1755      
=================================================================



Total params: 3,716,443
Trainable params: 3,716,443
Non-trainable params: 0

Found 22127 images belonging to 27 classes.
Found 11346 images belonging to 27 classes.

here is the code I used to make this model:

classifier = Sequential()

classifier.add(Convolution2D(32, (3, 3), input_shape=(sz, sz, 1), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))
classifier.add(Convolution2D(32, (3, 3), activation='relu'))
classifier.add(MaxPooling2D(pool_size=(2, 2)))

classifier.add(Flatten())

classifier.add(Dense(units=128,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=96,
                     activation='relu'))
classifier.add(Dropout(0.40))


classifier.add(Dense(units=64,
                     activation='relu'))

classifier.add(Dense(units=27, activation='softmax'))

optimizer used is adam and loss function I used is categorical_crossentropy.

from keras import optimizers

opt = optimizers.Adam(learning_rate = 0.001)

classifier.compile(optimizer=opt, loss='categorical_crossentropy', metrics=['accuracy'])

classifier.summary()

This is the code for flowing the data from the directories to the model

train_datagen = ImageDataGenerator(
        rescale=1./255,
        shear_range=0.2,
        zoom_range=0.2,
        horizontal_flip=True)

test_datagen = ImageDataGenerator(rescale=1./255)

training_set = train_datagen.flow_from_directory('Dataset2/Training Data',
                                                 target_size=(sz, sz),
                                                 batch_size=10,
                                                 color_mode='grayscale',
                                                 class_mode='categorical',
                                                 shuffle = True)

test_set = test_datagen.flow_from_directory('Dataset2/Test Data',
                                            target_size=(sz , sz),
                                            batch_size=10,
                                            color_mode='grayscale',
                                            class_mode='categorical',
                                            #shuffle = True
                                            )

classifier_obj = classifier.fit_generator(
        training_set,
        #steps_per_epoch=22127,
        epochs=10,
        validation_data=test_set,
        #validation_steps=11346
    verbose = 1)

the output this code produced was:

Epoch 1/10
2213/2213 [==============================] - 302s 136ms/step - loss: 0.0130 - accuracy: 0.7108 - val_loss: 0.0604 - val_accuracy: 0.3293
Epoch 2/10
2213/2213 [==============================] - 312s 141ms/step - loss: 0.0035 - accuracy: 0.9371 - val_loss: 0.0279 - val_accuracy: 0.4069
Epoch 3/10
2213/2213 [==============================] - 305s 138ms/step - loss: 0.0024 - accuracy: 0.9558 - val_loss: 0.0260 - val_accuracy: 0.4266
Epoch 4/10
2213/2213 [==============================] - 336s 152ms/step - loss: 0.0018 - accuracy: 0.9680 - val_loss: 0.0452 - val_accuracy: 0.4323
Epoch 5/10
2213/2213 [==============================] - 310s 140ms/step - loss: 0.0015 - accuracy: 0.9731 - val_loss: 0.0339 - val_accuracy: 0.3659
Epoch 6/10
2213/2213 [==============================] - 317s 143ms/step - loss: 0.0014 - accuracy: 0.9746 - val_loss: 0.0415 - val_accuracy: 0.4496
Epoch 7/10
2213/2213 [==============================] - 297s 134ms/step - loss: 0.0012 - accuracy: 0.9777 - val_loss: 0.0379 - val_accuracy: 0.4512
Epoch 8/10
2213/2213 [==============================] - 285s 129ms/step - loss: 0.0012 - accuracy: 0.9782 - val_loss: 0.0157 - val_accuracy: 0.4603
Epoch 9/10
2213/2213 [==============================] - 274s 124ms/step - loss: 0.0012 - accuracy: 0.9795 - val_loss: 0.0289 - val_accuracy: 0.4430
Epoch 10/10
2213/2213 [==============================] - 274s 124ms/step - loss: 9.1970e-04 - accuracy: 0.9837 - val_loss: 0.0459 - val_accuracy: 0.4800
Model Saved
Weights saved

Seeing promising results on this, I increased the number of epochs in hopes of getting a higher val_Accuracy but the result was that the val_accuracy was lower than the previous one.

I changed some parameters and ran the code with 100 epochs overnight. This was the output:

Some of the things that I found in other answers and have tried but failed:

1. used SGD optimizer
2. used regularizers
3. played around with the number of units in Dense layers.

Based on other answers, I think that either my model is overfitting or the data is bad. Please help me as I am completely clueless.

Answer 1

One of the major issue is the amount of records per categories. Definitely using data augmentation is always helpful to get good accuracy but much better approach is to use transfer learning techniques with data augmentation.

In your approach model is started overfitting because of that it's started having good training accuracy but bad validation accuracy. Let's have look into implementation of transfer learning. To understand more on transfer learning check this link Article Link

import tensorflow.keras as keras

model = keras.models.Sequential()

model.add(keras.applications.ResNet50(include_top=False, weights='imagenet', input_shape=(128, 128, 3)))

model.add(keras.layers.Flatten())

model.add(keras.layers.Dense(27, activation='softmax')) 

model.layers[0].trainable=False

model.compile(keras.optimizers.Adam(0.001), loss='categorical_crossentropy', metrics=['accuracy'])

model.summary()

model_obj = model.fit_generator(training_set, epochs=1, validation_data=test_set, verbose = 1)

Don't forgot to pass data augmented generator as a input data into fit_generator. There is no effect if you are using data augmentation on validation/test. You can use cross validation techniques or cnn based ensemble techniques for more good results.

Answer 2

This is overfitting, and it suggests that your images in each class are very similar to images across other classes.

Since your images across classes seem very similar, 800 per class is actually not a lot of data to train on. It's likely your model is struggling to discriminate the dev data into the correct classes based on what little it can learn., and your confusion matrix will likely confirm this.

I would try to get more data per class, maybe through augmentation, before trying any regularization techniques. To test this, you could try to run the model on far fewer classes first (say, 2) and see if its able to discriminate between these two better, if it can't then maybe some regularization is needed after all.

You coud also build more models to discriminate between fewer classes for each model, crucially, keeping the most confuseable classes into separate models.