I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:

1.0 0 0.4 0.6

Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:

0.8 0.2 0.6 0.4

is something like this:

1.0 0 0.8 0.2

Below you can see training + test loss/accuracy.

Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?

Training + testing Loss and accuracy


1 Answer 1


Seeing your graphs it does not show that the model is not learning, as the training accuracy is high. If the model wasn't learning anything even this would have been low. Also, you are right that the model is overfitting because it is performing very well on the training set and poorly on the test set.

The overfitting scenario is also confirmed by the fact that you have a very small dataset. You might want to fine-tune all the layers of the resnet as shown here. Also, try early stopping and dropout to prevent the model from overfitting.

  • $\begingroup$ How did you know I was learning only new layer :)? I know about dropouts but in pytorch I usually used it as layers. Is it even possible putting them in between layers in pretrained CNN? Is early stopping saving model? If it does then I sort of already do that but maybe I will use some proper library. $\endgroup$ Feb 8, 2019 at 13:00
  • $\begingroup$ Because that is the major reason for overfitting. No idea if we can put them in between layers but there might be some workaround to use it though. And yes, early stopping is saving your model when it stops getting better on validation set. $\endgroup$
    – bkshi
    Feb 8, 2019 at 13:08
  • $\begingroup$ Thank you. I will search how to add dropout layers to pretrained model in pytorch. $\endgroup$ Feb 8, 2019 at 13:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.