I am using TensorFlow to train a simple neural network (3 sequential dense layers). The problem is that the accuracy changes a lot every time I retrain it from scratch. I understand that, since the weights are initialized randomly, it may not always arrive at the exact same accuracy; but, I get a range of 4% for the accuracy on the test set.
This variation makes it impossible to check if different configurations of the network or different preprocessing steps for the data work better or worse because the configuration/preprocessing is a good/bad idea, or just because I got lucky/unlucky with the random initial weights.
This is an example of accuracies on 5 consecutive train+test's. The numbers are:
- Accuracy on the train set
- Accuracy on the validation split (20%)
- Accuracy on the test set
1. 2. 3. 4. Run #1: 95 91 74 20 Run #2: 94 92 75 18 Run #3: 94 91 74 20 Run #4: 94 92 73 20 Run #5: 94 91 77 17
I also find it confusing that there is no correlation between the accuracies for training, validation, and test sets.
I have tried different configurations of the ANN, longer trainings, shorter trainings, bigger and smaller validation split, different optimizers...nothing seems to give me a more stable accuracy among re-trainings. The numbers I have posted here are the best I could get.
Is a 4% range something normal? Is there a way to avoid those sub-optimal trainings? Could it be a problem related to local minima?