I'm currently training a CNN to do a binary classification. I'm getting fairly good results, but unfortunately the training is very unstable. Just by changing the seed the relative error changes by 20-30%.
What can be the cause of this and how can I prevent it?
- The amount of data is rather small.
- I'm starting from imagenet snapshot.