I have an image classification task, and I am using Keras for a network with CNN layers, with what seems like good results in training, translates to poor performance in testing. Upon training, I quickly see an increase in accuracy and validation accuracy to around the following level:
4678/4678 [==============================] - 2s 427us/step - loss: 0.0607 - acc: 0.9795 - val_loss: 0.1605 - val_acc: 0.9590
Now my first reaction was that this might be too good to be true, turns it was. In testing the model, it had very bad performance.
Bashar Haddad suggests that there may be a data imbalance problem, but with the following data specifications of data, before being split into 25% validation, I don't think that's my problem:
(data is being fit consecutively in a for loop) average number of datapoints in training: 4500 average number of datapoints in testing: 1500 average number of classes: 46 (max 49)
and most of the classes had a good amount of datapoints in each.
Based on this, I see the user found increasing batch size and decreasing learning rate helped, and I fit the model in the following way...
varying batch size from 100 -> 500 varying epochs size from 50 -> 250 cnn_model.fit(X_train, y_train, batch_size=500, epochs=50, verbose=1, shuffle=True, validation_data=(X_test, y_test) )
epochs had little to no effect.
With both accuracy and validation accuracy high, and also very little difference between them while training, I am confused as to why using this model on test data results in such bad performance.
What could be reasons for this? And possible solutions to solve it?