I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:

4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590

Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.

Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:

(data is being fit consecutively in a for loop)
average number of datapoints in training: 4500
average number of datapoints in testing: 1500
average number of classes: 46 (max 49)

and most of the classes had a good amount of datapoints in each.

Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way...

varying batch size from 100 -> 500
varying epochs size from 50 -> 250
cnn_model.fit(X_train, y_train,
                        validation_data=(X_test, y_test)

Varying batch_size and epochs had little to no effect.

With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.

What could be reasons for this? And possible solutions to solve it?

  • $\begingroup$ Obviously this has been a long time ago... but I'm wondering how did you achieve a batch size of 100-500, I can barely fit 5 images on my GPU 😄 I'm wondering, do you have a super-computer(cloud multiple GPU parallelism...?), tiny network (how many layers or params or node?)s, tiny image or some combinations of these? Or do you just have some black magic? $\endgroup$ – Elegant Code Mar 3 '20 at 0:55

You model might just overfit. This is a classical case with neural networks. It means that your model is very good on training data but performs poorly on testing data, i.e. it is very bad at generalizing...

You can monitor the performance on a validation set and use dropout to circumvent overfitting...

[EDIT] As @Jan van der Vegt said, overfitting on validation set is unlikely. Something similar happened to me in the past and it turned out that I was using a very small validation set. I'm still not sure why but I had very similar performance on both train and validation sets. As soon as I increased the size of the validation set, this behaviour disappeared. Although data leakage was indeed my first guess, I could not find any error in the way datasets were created. So my advice is: make sure that you don't have data leakage between training and validation set and make sure that your validation scores can be trusted by having a large enough validation size...

  • $\begingroup$ Overfitting on validation should not happen unless you do enormous parameter searches, I highly doubt this is the true issue $\endgroup$ – Jan van der Vegt Aug 22 '18 at 12:30
  • $\begingroup$ Thanks for the quick comment, instead of repeating myself, ill comment below. $\endgroup$ – jupiar Aug 23 '18 at 7:55
  • $\begingroup$ In addition, I was already using dropout, but also experimented by introducing more dropout layers throughout the model, to no effect. As for increasing the validationset, I will try that now, but it was already at 25%, something I thought already decent $\endgroup$ – jupiar Aug 23 '18 at 8:01
  • $\begingroup$ Is the problem still there? $\endgroup$ – qmeeus Aug 27 '18 at 8:20

I do not think this is an issue of overfitting. Your validation loss / accuracy is good and the domain does not lend itself very well for target leakage. My first inclination would be that the preprocessing before feeding your test images into the network is not the same, meaning your input distribution is different and the learned mapping is not applicable. Do you use training set statistics for the preprocessing of your images? Do you save this preprocessing pipeline somewhere and in the correct way?

Another issue could be that your train / validation split is not done properly, where the samples in your validation set are also in your training set, making it an improper validation set.

  • $\begingroup$ I have spent the better part of the day carefully going over the code, as there is alot, the preprocessing function is called by both training and testing, so no difference there, and the mapping of labels to vectors is saved as dictionary so everything that needs it, uses the same mapping. Training and validation split was done using one of sklearns built in functions: X_train, X_test, y_train, y_test = train_test_split(X, y, test_size= (1 - self.split_ratio), random_state=42), I have printouts of all these numpy shapes, and all look fine too, as posted in the question. $\endgroup$ – jupiar Aug 23 '18 at 7:59

I've had a very similar situation.

When I displayed images with plt.imshow()(matplotlib.pyplot.imshow() for the uninitiated 😄) They all looked perfectly fine, and I knew I wasn't doing any fancy pre-processing, so I figured my I was loading the data and "pre-processing" it just fine and identically for the train and test.

Long story short, I re-scaled my training data to [0,1] but I forgot to do the same for my test data (I was loading it with a slightly different method than I did my training data), so my test data remained in the range [1,255]. That resulted in very bad metrics on the test set.

plt.imshow() rescaled my data back so that I couldn't see the difference in scale between the images.

Moral of the story, if you exhausted all possible causes for meaningful differences between your validation set and test set (assuming CNN, and image classification, and no likely overfit on hyperparameters leading to data leakage from the validation set), check again, because you clearly didn't. There is still some cause for difference in the distribution of your train, validation , and test sets.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.