I'm using a shallow CNN for my current project [this one]. I have a training dataset consisting of 1000 samples and a test dataset of 400 samples. I'm using the test dataset to choose the best hyperparameters for the NN (the ones which give the smallest loss). The problem is that, when I repeat the execution (with the same hyperparameters) a few times, I obtain a very different loss each run.
There are two different sources of randomness that affect each run:
- The split of the training dataset in the different minibatches used for training
- The weights initialization of the network
The architecture of the CNN is the following one: a conv layer consisting of only two filters, a max pool layer, a fully connected layer with 16 neurons and the output layer, with only one unit. The activation function is ReLu.
My hypothesis is that my model depends a lot on the weights initialization (I'm currently using Xavier Initialization). I have tried using Dropout to prevent this high variance, which has helped to some extent, and also tested different batch sizes for training. My next step will be introducing batch normalization.
What else should I do to solve this problem?