0
$\begingroup$

I am trying to sanity-check my binary image classification model.
I am training it to overfit on 20 samples, now theoretically training loss should decrease and validation loss should increase. Because model should not be learning anything but both my train & val loss are decreasing.
Validation accuracy is also following a non-random pattern, Is my assertion for performance expectation true on overfitting on 20 samples and there is something wrong with my training loop/data loading process?
How can I triage it further?

enter image description here

$\endgroup$
5
  • 1
    $\begingroup$ How did you calculate the validation loss? On a left-out set or via cross validation? Are the training examples completely at random or they are somehow correlated? $\endgroup$
    – DaSim
    Aug 16, 2022 at 7:03
  • $\begingroup$ On a left-out set. Training examples are correlated, they are different textures of image which looks structurally same. $\endgroup$
    – Beginner
    Aug 16, 2022 at 13:08
  • 1
    $\begingroup$ Did you create those plots by hand? I think there is something wrong in them since the validation loss is always decreasing but the accuracy kinda stays the same (while it should decrease in order to have a lower loss) $\endgroup$
    – DaSim
    Aug 16, 2022 at 14:01
  • $\begingroup$ No, the validation loss is being fed to weights&biases(wandb.ai) API and the graph is generated from there but the metrics reported are in correspondence to the graph plotted $\endgroup$
    – Beginner
    Aug 16, 2022 at 14:54
  • $\begingroup$ Here is a much elaborated question if you want to have a look at further information - discuss.pytorch.org/t/… $\endgroup$
    – Beginner
    Aug 16, 2022 at 14:55

4 Answers 4

1
$\begingroup$

When training on a small sample, the network will be able to overfit to achieve perfect training loss. However, overfitting may not be required for achieving an optimal training loss. The premise that "theoretically training loss should decrease and validation loss should increase" is therefore not necessarily correct.

It may well be the case that there is a local loss optimum that classifies the sample correctly and still generalizes to your validation set. In case you are using some kind of sparsity penalty, that would make it even more likely that the network does not overfit. That said, a problem with the training loop is definitely in the cards given that loss and accuracy seem quite disconnected.

To triage further, I would recommend you manually inspect your training and validation dataset. Are you using a similarly small validation set? Are there classes in there that are not in the training set and could not possibly be learned? Are there multiple different classes in the training set or is it perhaps trivial to learn? Does training accuracy improve when training loss drops? How exactly does the validation loss drop without an increase in accuracy (check individually for some images, perhaps the model gets more confident for the ones it gets right but that should not lead to zero loss)?

$\endgroup$
1
  • $\begingroup$ The validation set is balanced(i.e equal number of instances of every class) while the training set is grossly imbalanced(50:1 ratio). It is a binary classification problem and I have reduced the size to 64*64 pixels to check errors in my training pipelines with speed. $\endgroup$
    – Beginner
    Aug 20, 2022 at 20:24
0
$\begingroup$

Suppose you have considered categoricalcrossentropy loss. It is roughly equivalent to -log(a) where a is activation value. Accuracy is percentage of correct classification. A change in activation value may lead to decrease/increase in loss but may not lead to increase/decrease in accuracy. Since you have taken a small number of samples, ensure that the samples have large variation. If one image is cat, take other image for a dog. Only miniscule changes like in texture , model will not have much to learn. If dataset has small variation, it may happen that validation loss decrease with epochs.

$\endgroup$
1
  • $\begingroup$ Let me clarify it further, when you say steps that means epoch right? Because I am training on 50 epochs here. $\endgroup$
    – Beginner
    Aug 16, 2022 at 13:27
0
$\begingroup$

In my opinion, your training and validation sets are almost same distributions and examples. Further more, now you are training the model with very few examples so it may not be overfitted.

To make the model overfitted,

  • You can try some more examples then use different distribution for training and validation sets.
  • Then you can set the learning rate as relatively small number.

Hope this will be helpful to your work.

$\endgroup$
0
$\begingroup$

Well if you look at the losses theoretically speaking "gradient is moving towards minimum but locally not globally". If you look at the val_accuracy graph first it goes up and then goes down then up then down it means you are stuck at the local minimum, therefore, the val_loss and train_loss are decreasing and the accuracy is bumping around. If you want to fix this issue then minimize your learning rate and if you are training only 20 images then take 10 as your batch size the noise on your graph is telling me that you have taken your batch size as 1 or 2.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.