I am trying to sanity-check my binary image classification model.
I am training it to overfit on 20 samples, now theoretically training loss should decrease and validation loss should increase. Because model should not be learning anything but both my train & val loss are decreasing.
Validation accuracy is also following a non-random pattern, Is my assertion for performance expectation true on overfitting on 20 samples and there is something wrong with my training loop/data loading process?
How can I triage it further?
When training on a small sample, the network will be able to overfit to achieve perfect training loss. However, overfitting may not be required for achieving an optimal training loss. The premise that "theoretically training loss should decrease and validation loss should increase" is therefore not necessarily correct.
It may well be the case that there is a local loss optimum that classifies the sample correctly and still generalizes to your validation set. In case you are using some kind of sparsity penalty, that would make it even more likely that the network does not overfit. That said, a problem with the training loop is definitely in the cards given that loss and accuracy seem quite disconnected.
To triage further, I would recommend you manually inspect your training and validation dataset. Are you using a similarly small validation set? Are there classes in there that are not in the training set and could not possibly be learned? Are there multiple different classes in the training set or is it perhaps trivial to learn? Does training accuracy improve when training loss drops? How exactly does the validation loss drop without an increase in accuracy (check individually for some images, perhaps the model gets more confident for the ones it gets right but that should not lead to zero loss)?
Suppose you have considered categoricalcrossentropy loss. It is roughly equivalent to -log(a) where a is activation value. Accuracy is percentage of correct classification. A change in activation value may lead to decrease/increase in loss but may not lead to increase/decrease in accuracy. Since you have taken a small number of samples, ensure that the samples have large variation. If one image is cat, take other image for a dog. Only miniscule changes like in texture , model will not have much to learn. If dataset has small variation, it may happen that validation loss decrease with epochs.
In my opinion, your training and validation sets are almost same distributions and examples. Further more, now you are training the model with very few examples so it may not be overfitted.
To make the model overfitted,
- You can try some more examples then use different distribution for training and validation sets.
- Then you can set the learning rate as relatively small number.
Hope this will be helpful to your work.
Well if you look at the losses theoretically speaking "gradient is moving towards minimum but locally not globally". If you look at the val_accuracy graph first it goes up and then goes down then up then down it means you are stuck at the local minimum, therefore, the val_loss and train_loss are decreasing and the accuracy is bumping around. If you want to fix this issue then minimize your learning rate and if you are training only 20 images then take 10 as your batch size the noise on your graph is telling me that you have taken your batch size as 1 or 2.