1
$\begingroup$

Note: Please go through this in its entirety.

My dataset:

  • A subset of the popular Dogs vs. Cats dataset (https://www.kaggle.com/c/dogs-vs-cats/data), made by me, because the original dataset had too many (25000 in 2 folders - cats and dogs - each with 12500) labeled images in the training folder and randomly arranged unlabelled images in the testing folder.
  • My dataset named 'dogs-vs-cats_15000_AbC', with 15000 images, has 3 folders:
  • Training folder: my-train-AbC_9000_1st-from-'train' - 9000 images - 4500 cats & 4500 dogs
  • Validation folder: my-valid-AbC_3000_2nd-from-'train' - 3000 images - 1500 cats & 1500 dogs
  • Testing folder: my-test-AbC_3000_3rd-from-'train' - 3000 images - 1500 cats & 1500 dogs
  • There is thus a 60:20:20 split.
  • No images overlap.

Highlights:

  • img_height, img_width = 150, 150 fixed. Note that images come in various dimensions originally. Upscaling downscaling going on underneath.
  • Data Augmentation was done and 3 data generators created:
 train_generator = train_datagen.flow_from_directory(train_data_dir, ...

 valid_generator = train_datagen.flow_from_directory(valid_data_dir, ...

 test_generator = test_datagen.flow_from_directory(test_data_dir, ...
  • Notice how I got a very high train and valid accuracies very quickly such that the latter is higher than the former:

Epoch 1/100 90/90 [==============================] - 6157s 68s/step - loss: 0.4265 - accuracy: 0.9142 - val_loss: 0.1147 - val_accuracy: 0.9530

...

Epoch 15/100 90/90 [==============================] - 86s 954ms/step - loss: 0.0282 - accuracy: 0.9907 - val_loss: 0.0992 - val_accuracy: 0.9650

  • Now look at the testing accuracy which varies across 10 iterations and is generally poor:
for i in range(1, 11):
  evaluate = save_bottlebeck_features(None).evaluate(test_generator, steps = test_generator.n // batch_size, verbose =1)
  print('Accuracy Test : {}'.format(evaluate[1]))

Model loaded.
30/30 [==============================] - 12s 335ms/step - loss: 1.5205 - accuracy: 0.5470

Accuracy Test : 0.546999990940094

Thank you for your time.

$\endgroup$
6
  • 1
    $\begingroup$ What steps are you taking to prevent overfitting ? $\endgroup$
    – lcrmorin
    Jun 21 at 19:01
  • $\begingroup$ One possibility is that the testing set would have a different distribution than the validation set (This could be excluded by joining all the data, randomizing, and splitting again to train, test, valid). $\endgroup$ Jun 23 at 15:06
  • $\begingroup$ To swap valid and test with each other and see if it has an effect (Sometimes if one set has relatively harder examples). $\endgroup$ Jun 23 at 15:06
  • $\begingroup$ If the training somehow overfitted on the validation set (Is it possible that during training, at one or more steps, the model giving the best score on the validation set is chosen). $\endgroup$ Jun 23 at 15:06
  • $\begingroup$ Images overlapping, lack of shuffling, data-augmentation methods not suitable enough. $\endgroup$ Jun 23 at 15:07
1
$\begingroup$

Your training accuracy seems too high, which could lead to overfitting, i.e. poor generalisation on new/test data.

You should add some functions like a dropout (~0.1) to improve generalisation and have a good training result (about 90-95%).

$\endgroup$
1
  • $\begingroup$ Thank you for your response! Yes, that is indeed a valid point. However, in my case, there was actually a bug in my code. I had defined my InceptionV3 model inside a function that was being trained upon and validated upon as usual. However, during evaluation, the way that I had coded made it create a new instance every time during testing which was why the testing accuracy was poor. After correcting my error, I managed to get the desired testing accuracy of 96%+. You can check out the notebook if you'd like. $\endgroup$ Jun 23 at 6:53
0
$\begingroup$

The problem has been resolved. The datasets and the model were fine in my case. However, there was actually a bug in the code, more specifically in the evaluation cell.

I had defined my InceptionV3 model inside a function that was being trained upon and validated upon as usual. However, during evaluation, the way that I had coded made it create a new model instance every time during testing which was why the testing accuracy was poor.

After correcting my error, I managed to get the desired testing accuracy of 96%+. You can check out the notebook if you'd like after the update. I think this may be the issue in the case of others too, assuming their datasets are well made. Generally, in Deep Learning, if something is too odd or even too good to be true, it may have something to do with a bug in the code.

$\endgroup$
2
  • 1
    $\begingroup$ This is an interesting cautionary tale - there is nothing in your question that could possibly point someone answering it to this bug. The best someone could say from what you asked is "maybe you have a bug in your implementation". $\endgroup$ Jun 23 at 7:14
  • $\begingroup$ @NeilSlater Yes, I agree. I had posted the same in other forums too and one person who had actually looked into the accompanying Colab file was finally able to resolve my query. Believing myself to be a newbie in Transfer Learning applications, I was looking in all the wrong places, whereas the actual error had nothing to do with TF but was actually about functions and objects in python. $\endgroup$ Jun 23 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.