I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:
Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:
is something like this:
Below you can see training + test loss/accuracy.
Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?