Validation acc is very high in each fold but Test acc is very low

Question

I am trying to implement a neural network. I am using CNN model for classifying. First I split the dataset into train and test.

Code Snippet:

X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42, shuffle=True, stratify=Y)

then I built a CNN model and used stratified cross-validation to fit the model.

Code Snippet:

from statistics import mean, stdev
# Loop through the splits
lst_accu_stratified = []
for train_index, val_index in skf.split(X_train, y_train):
    X_train_fold, X_val_fold = X_train[train_index], X_train[val_index]
    y_train_fold, y_val_fold = y_train[train_index], y_train[val_index]
    # print('Fold :')
    ResNet50 = model.fit(X_train_fold, y_train_fold, batch_size=16, epochs=20, verbose=1)
    val_loss, val_acc = model.evaluate(X_val_fold, y_val_fold, verbose=0)
    print("Validation Loss: ", val_loss, "Validation Accuracy: ", val_acc)
    lst_accu_stratified.append(val_acc)

# Print the output.
print('List of possible accuracy:', lst_accu_stratified)
print('\nMaximum Accuracy That can be obtained from this model is:',
      max(lst_accu_stratified)*100, '%')
print('\nMinimum Accuracy:',
      min(lst_accu_stratified)*100, '%')
print('\nOverall Accuracy:',
      mean(lst_accu_stratified)*100, '%')
print('\nStandard Deviation is:', stdev(lst_accu_stratified))

Output:

Epoch 1/20
30/30 [==============================] - 9s 102ms/step - loss: 1.3490 - accuracy: 0.5756
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4620 - accuracy: 0.8466
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1818 - accuracy: 0.9412
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1106 - accuracy: 0.9727
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0643 - accuracy: 0.9811
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0438 - accuracy: 0.9895
Epoch 7/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0371 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0212 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0143 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0149 - accuracy: 0.9958
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0158 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0134 - accuracy: 0.9958
Epoch 13/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0072 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0031 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0019 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0088 - accuracy: 0.9979
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0031 - accuracy: 1.0000
Validation Loss:  0.8360670208930969 Validation Accuracy:  0.800000011920929
Epoch 1/20
30/30 [==============================] - 3s 106ms/step - loss: 0.5129 - accuracy: 0.8700
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4789 - accuracy: 0.8784
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.2724 - accuracy: 0.9224
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.2108 - accuracy: 0.9308
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1081 - accuracy: 0.9706
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1010 - accuracy: 0.9748
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0481 - accuracy: 0.9895
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0316 - accuracy: 0.9874
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0483 - accuracy: 0.9811
Epoch 10/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0167 - accuracy: 0.9937
Epoch 11/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0129 - accuracy: 0.9937
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0093 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0389 - accuracy: 0.9895
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0293 - accuracy: 0.9895
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7058e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0011 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7595e-04 - accuracy: 1.0000
Validation Loss:  0.5674645304679871 Validation Accuracy:  0.8571428656578064
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1533 - accuracy: 0.9518
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0978 - accuracy: 0.9686
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0702 - accuracy: 0.9790
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0754 - accuracy: 0.9811
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0362 - accuracy: 0.9874
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0174 - accuracy: 0.9916
Epoch 7/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0144 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0089 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0017 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0044 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0033 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 5.9884e-04 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 3.7875e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 4.7657e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 2.8062e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 4.5594e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 2.3471e-04 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 2.5190e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 1.5143e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 2.4174e-04 - accuracy: 1.0000
Validation Loss:  0.002929181093350053 Validation Accuracy:  1.0
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0035 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0048 - accuracy: 0.9979
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 7.1234e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0100 - accuracy: 0.9937
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0041 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 6.2473e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 4.5511e-04 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0015 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0132 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0106 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0032 - accuracy: 0.9979
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 0.9979
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0039 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 2.7678e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0034 - accuracy: 0.9979
Epoch 19/20
30/30 [==============================] - 2s 73ms/step - loss: 4.1879e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 4.0388e-04 - accuracy: 1.0000
Validation Loss:  0.003368004923686385 Validation Accuracy:  1.0
Epoch 1/20
30/30 [==============================] - 2s 72ms/step - loss: 5.1283e-04 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 72ms/step - loss: 8.4923e-04 - accuracy: 1.0000
Epoch 3/20
30/30 [==============================] - 2s 72ms/step - loss: 3.2774e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 1.3468e-04 - accuracy: 1.0000
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 1.4561e-04 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 1.6656e-04 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 1.2794e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7647e-05 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7325e-04 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 6.5071e-05 - accuracy: 1.0000
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 6.1966e-05 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 5.9293e-05 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 3.1360e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 1.0051e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7242e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 5.6384e-05 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 8.4639e-05 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7929e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 1.6557e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 4.6414e-04 - accuracy: 1.0000
Validation Loss:  8.931908087106422e-05 Validation Accuracy:  1.0
List of possible accuracy: [0.800000011920929, 0.8571428656578064, 1.0, 1.0, 1.0]

Maximum Accuracy That can be obtained from this model is: 100.0 %

Minimum Accuracy: 80.0000011920929 %

Overall Accuracy: 93.1428575515747 %

Standard Deviation is: 0.09604420178372833

here the val accuracy of each fold is pretty high but when I test the model with test dataset, the accuracy is very low.

Code snippet:

model.evaluate(X_test, y_test,batch_size=32)

output:

5/5 [==============================] - 1s 222ms/step - loss: 2.3315 - accuracy: 0.6913
[2.3314528465270996, 0.6912751793861389]

My question is,

Is my method correct?
What can be the reason for low test accuracy?

@RobinvanHoorn created the stratified k-fold cross-validator skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42) — Rezuana Haque, Jan 24 at 12:58

shepan6 · Accepted Answer · 2023-01-24 10:53:57Z

thanks for the detailed question here, which asks about 1) overall methodology and reasoning for low test accuracy.
1) Overall, the methodology looks sound from what is described in the post.
2) For checking for reasons behind low test accuracy, I would review the following:

I would check for class imbalance (as I don't currently see the use of class weights in the implementation above) among the training, validation and test sets. For example if the training set has 90% Class A and 10% Class B (and similar for validation), and the test set has more 50:50 distribution, then of course low accuracy is to be expected.
I would also check for overfitting as well, just in case the model has fit well to the noise of the data and the validation data is similar in terms of distributions of input features. This can be done by looking at training and validation loss over epochs for each fold.
I would check to see if the test data differs significantly from the training and validation data. For this, you can statistical testing (e.g. ANOVA or t-test) over each input feature distribution to check if this is the case or not.

Hope this helps!

Thank you for your response @shepan6 1. The dataset contains class imbalance. But I used Stratified k-fold validation which can help improve the performance of a model when dealing with class imbalance. 2. there is overfitting I think. But can you please tell me how can I deal with this? I tried to use early stopping but there is error tensorflow:Early stopping conditioned on metric val_loss` which is not available. Available metrics are: loss,accuracy` 3. Can you please provide a code sample or link or blog, anything you can? — Rezuana Haque, Jan 27 at 7:10
In regards to combatting overfitting: This can be done by: reducing model complexity (removing layers in the model, overall reducing number of parameters [weights, biases, etc.]), using data augmentation to artificially create more examples which reflect the distribution of the dataset. To include the possibility of evaluating validation statistics over epochs, you need to add the validation_data or validation_split parameter to the model.fit() function (tensorflow.org/guide/keras/…) — shepan6, Feb 7 at 10:46

Stack Exchange Network

Validation acc is very high in each fold but Test acc is very low

1 Answer 1

Your Answer

Not the answer you're looking for? Browse other questions tagged
python
deep-learning
keras
tensorflow
cnn
or ask your own question.

Hot Network Questions

Validation acc is very high in each fold but Test acc is very low

1 Answer 1

Your Answer

Sign up or log in

Post as a guest

Not the answer you're looking for? Browse other questions tagged pythondeep-learningkerastensorflowcnn or ask your own question.

Related

Hot Network Questions

Not the answer you're looking for? Browse other questions tagged
python
deep-learning
keras
tensorflow
cnn
or ask your own question.