There are a couple of things to know around this topic:
It might be difficult to get identical results using Keras. This is because it is a wrapper around lower-level libraries, such as Tensorflow, Theano and CNTK.
Using these backends, a static graph is built that represent all the computation steps in your network. This then allows automatic differentiation (and so backpropagation) to be performed. The graph that is built may be in several blocks. For example, in Tensorflow, you can use context managers to separate when and how weights are updated (essentially using
If your model does have these (under the hood or otherwise!), you would need to set a random seed in each of those block. You can a little more on this topic, here.
In addition to the above, there are operators in Tensorflow (and probably other frameworks), which use approximations/simplifications for the sake of efficiency and so speed.
tf.reduce_sum is an example that introduces non-deterministic deviations that could lead to your variations in accuracy. This operator is used to add up the errors of your model and will do so in a parallelised way where we cannot know the order (or set it with a seed). The problem arises because summation of numbers, as used in that operator, are not commutative.
If I add the numbers
1 + 2 + 7 or the numbers
7 + 1 + 2, both give us the result of 10 - because addition is commutative. However, in floating point addition, where we are adding numbers like
1.2223427 + 7.0195516 + 1.9719819, (or actually numbers with much more decimal places) there will be a loss in accuracy as we cannot retain all information... one could imagine it like rounding errors. It is also referred to as catastrophic cancellation. See more details here.
In this case, the order in which we add up the numbers will matter! As I mentioned earlier, the parallelisation of operations will mean that we cannot know the order of the operations, and so we cannot guarantee identical answers for the same runs of an algorithm, while still enjoying the parallel computations!
Although this might cause a headache for some people, because reproducibility is a big - both in academic research as well as industry applications - the variation in results due to this pseudo-randomness and parallelisation/summation errors is really negligible in the bigger picture.
Changing a layer in a deep NN, altering the learning rate or the regularisation are all factors that are much more important and will make larger differences in results. They also encode knowledge and decisions made by you, as a practitioner. I would suggest spending time thinking about these things and not at all worry about these small blips.
There is a nice post from Python Guru & Core DEv: Raymond Hettinger, where he shows how to maintain full precision for summations of floating point numbers. It involves keeping track of sub-totals, which can be used to ensure the final sum did not cause any loss of precision.