This is an issue I've been encountering before and I was wondering what can be potential causes for this. Occasionally, training of an identical setup LSTM model ( using Keras ), on the same training data, results in a much greater achieved validation accuracy.
E.g., a common scenario would be out of 10 training runs, it'd achieve a validation accuracy of 63% 8 out of 10 times, and 79% 2 out of 10 times. Out of all independent training runs it once even achieved a validation accuracy of 91%.
I know this can be caused by randomness in the neural network, I have (20%) dropout introduced and by default the seeds are randomly initialized by keras. However, is there a way I can "ensure" my model will find its best solution in one training? Or at least make it more reliable? Because it seems to indicate that my model gets stuck in local optima as it doesn't always find better solutions, even though it aparantly is able to find them in other runs.
Setting the seed to a constant value would not solve the issue because it will just force it to just find that 1 solution that corresponds with that specific seed, which may not be the best it's able to find in general.
Currently, I'm just training the model n times to assess its performance, analyzing its average and highest validation accuracy obtained. However this slows down my experiments in finding the best model a lot, and also the reliability is unknown.
So in short, how can I improve my model's ability to find its best solution in one training? Or what could be potential causes for it being inconsistent in doing so?
Any insights on this would be much appreciated!