Seemingly good results with training a CNN but bad when testing

Question

I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:

4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590

Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.

Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:

(data is being fit consecutively in a for loop)
average number of datapoints in training: 4500
average number of datapoints in testing: 1500
average number of classes: 46 (max 49)

and most of the classes had a good amount of datapoints in each.

Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way...

varying batch size from 100 -> 500
varying epochs size from 50 -> 250
cnn_model.fit(X_train, y_train,
                        batch_size=500,
                        epochs=50,
                        verbose=1,
                        shuffle=True,
                        validation_data=(X_test, y_test)
            )

Varying batch_size and epochs had little to no effect.

With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.

What could be reasons for this? And possible solutions to solve it?

Answer 1

You model might just overfit. This is a classical case with neural networks. It means that your model is very good on training data but performs poorly on testing data, i.e. it is very bad at generalizing...

You can monitor the performance on a validation set and use dropout to circumvent overfitting...

[EDIT] As @Jan van der Vegt said, overfitting on validation set is unlikely. Something similar happened to me in the past and it turned out that I was using a very small validation set. I'm still not sure why but I had very similar performance on both train and validation sets. As soon as I increased the size of the validation set, this behaviour disappeared. Although data leakage was indeed my first guess, I could not find any error in the way datasets were created. So my advice is: make sure that you don't have data leakage between training and validation set and make sure that your validation scores can be trusted by having a large enough validation size...

Answer 2

I do not think this is an issue of overfitting. Your validation loss / accuracy is good and the domain does not lend itself very well for target leakage. My first inclination would be that the preprocessing before feeding your test images into the network is not the same, meaning your input distribution is different and the learned mapping is not applicable. Do you use training set statistics for the preprocessing of your images? Do you save this preprocessing pipeline somewhere and in the correct way?

Another issue could be that your train / validation split is not done properly, where the samples in your validation set are also in your training set, making it an improper validation set.

Answer 3

I've had a very similar situation.

When I displayed images with plt.imshow()(matplotlib.pyplot.imshow() for the uninitiated 😄) They all looked perfectly fine, and I knew I wasn't doing any fancy pre-processing, so I figured my I was loading the data and "pre-processing" it just fine and identically for the train and test.

Long story short, I re-scaled my training data to [0,1] but I forgot to do the same for my test data (I was loading it with a slightly different method than I did my training data), so my test data remained in the range [1,255]. That resulted in very bad metrics on the test set.

plt.imshow() rescaled my data back so that I couldn't see the difference in scale between the images.

Moral of the story, if you exhausted all possible causes for meaningful differences between your validation set and test set (assuming CNN, and image classification, and no likely overfit on hyperparameters leading to data leakage from the validation set), check again, because you clearly didn't. There is still some cause for difference in the distribution of your train, validation , and test sets.