I am trying to implement a neural network. I am using CNN model for classifying. First I split the dataset into train and test.
Code Snippet:
X_train, X_test, y_train, y_test = train_test_split(X, Y, test_size=0.2, random_state=42, shuffle=True, stratify=Y)
then I built a CNN model and used stratified cross-validation to fit the model.
Code Snippet:
from statistics import mean, stdev
# Loop through the splits
lst_accu_stratified = []
for train_index, val_index in skf.split(X_train, y_train):
X_train_fold, X_val_fold = X_train[train_index], X_train[val_index]
y_train_fold, y_val_fold = y_train[train_index], y_train[val_index]
# print('Fold :')
ResNet50 = model.fit(X_train_fold, y_train_fold, batch_size=16, epochs=20, verbose=1)
val_loss, val_acc = model.evaluate(X_val_fold, y_val_fold, verbose=0)
print("Validation Loss: ", val_loss, "Validation Accuracy: ", val_acc)
lst_accu_stratified.append(val_acc)
# Print the output.
print('List of possible accuracy:', lst_accu_stratified)
print('\nMaximum Accuracy That can be obtained from this model is:',
max(lst_accu_stratified)*100, '%')
print('\nMinimum Accuracy:',
min(lst_accu_stratified)*100, '%')
print('\nOverall Accuracy:',
mean(lst_accu_stratified)*100, '%')
print('\nStandard Deviation is:', stdev(lst_accu_stratified))
Output:
Epoch 1/20
30/30 [==============================] - 9s 102ms/step - loss: 1.3490 - accuracy: 0.5756
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4620 - accuracy: 0.8466
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1818 - accuracy: 0.9412
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1106 - accuracy: 0.9727
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0643 - accuracy: 0.9811
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0438 - accuracy: 0.9895
Epoch 7/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0371 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0212 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0143 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0149 - accuracy: 0.9958
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0158 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0134 - accuracy: 0.9958
Epoch 13/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0072 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0031 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0019 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0088 - accuracy: 0.9979
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0031 - accuracy: 1.0000
Validation Loss: 0.8360670208930969 Validation Accuracy: 0.800000011920929
Epoch 1/20
30/30 [==============================] - 3s 106ms/step - loss: 0.5129 - accuracy: 0.8700
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.4789 - accuracy: 0.8784
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.2724 - accuracy: 0.9224
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.2108 - accuracy: 0.9308
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1081 - accuracy: 0.9706
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1010 - accuracy: 0.9748
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0481 - accuracy: 0.9895
Epoch 8/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0316 - accuracy: 0.9874
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0483 - accuracy: 0.9811
Epoch 10/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0167 - accuracy: 0.9937
Epoch 11/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0129 - accuracy: 0.9937
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0024 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0093 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0389 - accuracy: 0.9895
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0293 - accuracy: 0.9895
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7058e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0011 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7595e-04 - accuracy: 1.0000
Validation Loss: 0.5674645304679871 Validation Accuracy: 0.8571428656578064
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.1533 - accuracy: 0.9518
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0978 - accuracy: 0.9686
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0702 - accuracy: 0.9790
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0754 - accuracy: 0.9811
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0362 - accuracy: 0.9874
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0174 - accuracy: 0.9916
Epoch 7/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0144 - accuracy: 0.9916
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0089 - accuracy: 0.9958
Epoch 9/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0017 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0044 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0033 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 5.9884e-04 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 3.7875e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 4.7657e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 2.8062e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 4.5594e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 72ms/step - loss: 2.3471e-04 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 72ms/step - loss: 2.5190e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 72ms/step - loss: 1.5143e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 2.4174e-04 - accuracy: 1.0000
Validation Loss: 0.002929181093350053 Validation Accuracy: 1.0
Epoch 1/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0035 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0048 - accuracy: 0.9979
Epoch 3/20
30/30 [==============================] - 2s 71ms/step - loss: 7.1234e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0100 - accuracy: 0.9937
Epoch 5/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0041 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 71ms/step - loss: 0.0016 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 6.2473e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 72ms/step - loss: 4.5511e-04 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0015 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0132 - accuracy: 0.9979
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 0.0106 - accuracy: 0.9958
Epoch 12/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0032 - accuracy: 0.9979
Epoch 13/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 0.9979
Epoch 14/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0039 - accuracy: 0.9979
Epoch 15/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0023 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 73ms/step - loss: 2.7678e-04 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0022 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 73ms/step - loss: 0.0034 - accuracy: 0.9979
Epoch 19/20
30/30 [==============================] - 2s 73ms/step - loss: 4.1879e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 72ms/step - loss: 4.0388e-04 - accuracy: 1.0000
Validation Loss: 0.003368004923686385 Validation Accuracy: 1.0
Epoch 1/20
30/30 [==============================] - 2s 72ms/step - loss: 5.1283e-04 - accuracy: 1.0000
Epoch 2/20
30/30 [==============================] - 2s 72ms/step - loss: 8.4923e-04 - accuracy: 1.0000
Epoch 3/20
30/30 [==============================] - 2s 72ms/step - loss: 3.2774e-04 - accuracy: 1.0000
Epoch 4/20
30/30 [==============================] - 2s 72ms/step - loss: 1.3468e-04 - accuracy: 1.0000
Epoch 5/20
30/30 [==============================] - 2s 72ms/step - loss: 1.4561e-04 - accuracy: 1.0000
Epoch 6/20
30/30 [==============================] - 2s 72ms/step - loss: 1.6656e-04 - accuracy: 1.0000
Epoch 7/20
30/30 [==============================] - 2s 71ms/step - loss: 1.2794e-04 - accuracy: 1.0000
Epoch 8/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7647e-05 - accuracy: 1.0000
Epoch 9/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7325e-04 - accuracy: 1.0000
Epoch 10/20
30/30 [==============================] - 2s 72ms/step - loss: 6.5071e-05 - accuracy: 1.0000
Epoch 11/20
30/30 [==============================] - 2s 72ms/step - loss: 6.1966e-05 - accuracy: 1.0000
Epoch 12/20
30/30 [==============================] - 2s 71ms/step - loss: 5.9293e-05 - accuracy: 1.0000
Epoch 13/20
30/30 [==============================] - 2s 71ms/step - loss: 3.1360e-04 - accuracy: 1.0000
Epoch 14/20
30/30 [==============================] - 2s 71ms/step - loss: 1.0051e-04 - accuracy: 1.0000
Epoch 15/20
30/30 [==============================] - 2s 71ms/step - loss: 1.7242e-04 - accuracy: 1.0000
Epoch 16/20
30/30 [==============================] - 2s 71ms/step - loss: 5.6384e-05 - accuracy: 1.0000
Epoch 17/20
30/30 [==============================] - 2s 71ms/step - loss: 8.4639e-05 - accuracy: 1.0000
Epoch 18/20
30/30 [==============================] - 2s 71ms/step - loss: 6.7929e-04 - accuracy: 1.0000
Epoch 19/20
30/30 [==============================] - 2s 71ms/step - loss: 1.6557e-04 - accuracy: 1.0000
Epoch 20/20
30/30 [==============================] - 2s 71ms/step - loss: 4.6414e-04 - accuracy: 1.0000
Validation Loss: 8.931908087106422e-05 Validation Accuracy: 1.0
List of possible accuracy: [0.800000011920929, 0.8571428656578064, 1.0, 1.0, 1.0]
Maximum Accuracy That can be obtained from this model is: 100.0 %
Minimum Accuracy: 80.0000011920929 %
Overall Accuracy: 93.1428575515747 %
Standard Deviation is: 0.09604420178372833
here the val accuracy of each fold is pretty high but when I test the model with test dataset, the accuracy is very low.
Code snippet:
model.evaluate(X_test, y_test,batch_size=32)
output:
5/5 [==============================] - 1s 222ms/step - loss: 2.3315 - accuracy: 0.6913
[2.3314528465270996, 0.6912751793861389]
My question is,
- Is my method correct?
- What can be the reason for low test accuracy?
skf.split
? $\endgroup$skf = StratifiedKFold(n_splits=5, shuffle=True, random_state=42)
$\endgroup$