I am using Keras in Jupyter Notebook.
I understood that for the same results, the random numbers should be produced from the same seed each time.
So, in the first of all my codes, I set random.seed as 1234 in a cell.
np.random.seed(1234)
Then other cells are the code for my model and the fit and evaluate code. But each time that I run the model cells, the loss values are different!
Why does it happen? How can I solve it?
It's not quite enough to set only the numpy random seed, as you've seen - the Keras documentation also notes that it's necessary to set:
The interaction of all of these will generally result in different loss values due mainly to changes in random weight initialisation, which data ends up in your train/val/test splits, and the order that data is passed to your model for training.
As others have mentioned, a small amount of variance is to be expected and it's certainly not worth hamstringing your performance over (by limiting yourself to a single thread); setting the seeds and rng should be enough to satisfy people your results are reproducible.
If your results still have unacceptable variance after taking those steps then it might be an indication that your model is brittle and may not generalise well to new data so you'll want to address that.
Final thought - when you say the values are different each time you run the model cell, are you deleting/overwriting the model you already trained? If you are running the model cell on existing model/weight values then that's the same as training for more epochs and would usually have a large impact on your results.
import random from numpy.random import seed from tensorflow import set_random_seed set_random_seed(2) seed(1) random.seed(1) should I use these codes before model creation codes every time?
$\endgroup$
Commented
Aug 25, 2018 at 14:59
There are a couple of things to know around this topic:
It might be difficult to get identical results using Keras. This is because it is a wrapper around lower-level libraries, such as Tensorflow, Theano and CNTK.
Using these backends, a static graph is built that represent all the computation steps in your network. This then allows automatic differentiation (and so backpropagation) to be performed. The graph that is built may be in several blocks. For example, in Tensorflow, you can use context managers to separate when and how weights are updated (essentially using with blocks.
If your model does have these (under the hood or otherwise!), you would need to set a random seed in each of those block. You can a little more on this topic, here.
In addition to the above, there are operators in Tensorflow (and probably other frameworks), which use approximations/simplifications for the sake of efficiency and so speed. tf.reduce_sum is an example that introduces non-deterministic deviations that could lead to your variations in accuracy. This operator is used to add up the errors of your model and will do so in a parallelised way where we cannot know the order (or set it with a seed). The problem arises because summation of numbers, as used in that operator, are not commutative.
Example:
If I add the numbers 1 + 2 + 7 or the numbers 7 + 1 + 2, both give us the result of 10 - because addition is commutative. However, in floating point addition, where we are adding numbers like 1.2223427 + 7.0195516 + 1.9719819, (or actually numbers with much more decimal places) there will be a loss in accuracy as we cannot retain all information... one could imagine it like rounding errors. It is also referred to as catastrophic cancellation. See more details here.
In this case, the order in which we add up the numbers will matter! As I mentioned earlier, the parallelisation of operations will mean that we cannot know the order of the operations, and so we cannot guarantee identical answers for the same runs of an algorithm, while still enjoying the parallel computations!
Although this might cause a headache for some people, because reproducibility is a big - both in academic research as well as industry applications - the variation in results due to this pseudo-randomness and parallelisation/summation errors is really negligible in the bigger picture.
Changing a layer in a deep NN, altering the learning rate or the regularisation are all factors that are much more important and will make larger differences in results. They also encode knowledge and decisions made by you, as a practitioner. I would suggest spending time thinking about these things and not at all worry about these small blips.
There is a nice post from Python Guru & Core DEv: Raymond Hettinger, where he shows how to maintain full precision for summations of floating point numbers. It involves keeping track of sub-totals, which can be used to ensure the final sum did not cause any loss of precision.
The results (loss after a certain number of epochs) will be the same every time if you initialize the pseudorandom number generator (np.random.seed(1234)) before importing Keras and restart the Python interpreter between the runs.
