# Transfer learning - small database

I am trying to use transfer learning in medical (ultrasound pictures). The problem is - I have very limited picture database = 400 (360+40). I am using resnet50 (I don't think this is important but maybe I'm wrong). Resnet as feature extractor + SVM is not great but normalized confusion matrix is somewhat about:

1.0 0 0.4 0.6

Now, I wanted to fine-tune resnet. And the problem is that CM at the beginning looks like:

0.8 0.2 0.6 0.4

is something like this:

1.0 0 0.8 0.2

Below you can see training + test loss/accuracy.

Now I thought it is overfitting (due to too large rate capacity / database) but someone pointed that network might not be learning. What is the case?