4
$\begingroup$

I am using Keras in Jupyter Notebook.

I understood that for the same results, the random numbers should be produced from the same seed each time.

So, in the first of all my codes, I set random.seed as 1234 in a cell.

np.random.seed(1234)

Then other cells are the code for my model and the fit and evaluate code. But each time that I run the model cells, the loss values are different!

Why does it happen? How can I solve it?

$\endgroup$
  • $\begingroup$ If the difference isn't too much, then it's acceptable.. $\endgroup$ – Aditya Aug 25 '18 at 7:44
3
$\begingroup$

It's not quite enough to set only the numpy random seed, as you've seen - the Keras documentation also notes that it's necessary to set:

  • the python hash seed
  • the core python rng
  • the random seed of your backend to keras
  • and force your backend to use a single thread.

The interaction of all of these will generally result in different loss values due mainly to changes in random weight initialisation, which data ends up in your train/val/test splits, and the order that data is passed to your model for training.

As others have mentioned, a small amount of variance is to be expected and it's certainly not worth hamstringing your performance over (by limiting yourself to a single thread); setting the seeds and rng should be enough to satisfy people your results are reproducible.

If your results still have unacceptable variance after taking those steps then it might be an indication that your model is brittle and may not generalise well to new data so you'll want to address that.

Final thought - when you say the values are different each time you run the model cell, are you deleting/overwriting the model you already trained? If you are running the model cell on existing model/weight values then that's the same as training for more epochs and would usually have a large impact on your results.

$\endgroup$
  • $\begingroup$ Thank you very much. The Tensorflow is the backend of Keras. I used these codes, it produces approximately the same results for each running. import random from numpy.random import seed from tensorflow import set_random_seed set_random_seed(2) seed(1) random.seed(1) should I use these codes before model creation codes every time? $\endgroup$ – user7194905 Aug 25 '18 at 14:59
  • $\begingroup$ check out the link to the keras docs I posted, it's got code snippets for everything so you can decide which elements you'd like to use $\endgroup$ – redhqs Aug 26 '18 at 8:33
0
$\begingroup$

There are a couple of things to know around this topic:

Keras Backends

It might be difficult to get identical results using Keras. This is because it is a wrapper around lower-level libraries, such as Tensorflow, Theano and CNTK.

Using these backends, a static graph is built that represent all the computation steps in your network. This then allows automatic differentiation (and so backpropagation) to be performed. The graph that is built may be in several blocks. For example, in Tensorflow, you can use context managers to separate when and how weights are updated (essentially using with blocks.

If your model does have these (under the hood or otherwise!), you would need to set a random seed in each of those block. You can a little more on this topic, here.

Catastrophic cancellation

In addition to the above, there are operators in Tensorflow (and probably other frameworks), which use approximations/simplifications for the sake of efficiency and so speed. tf.reduce_sum is an example that introduces non-deterministic deviations that could lead to your variations in accuracy. This operator is used to add up the errors of your model and will do so in a parallelised way where we cannot know the order (or set it with a seed). The problem arises because summation of numbers, as used in that operator, are not commutative.

Example:

If I add the numbers 1 + 2 + 7 or the numbers 7 + 1 + 2, both give us the result of 10 - because addition is commutative. However, in floating point addition, where we are adding numbers like 1.2223427 + 7.0195516 + 1.9719819, (or actually numbers with much more decimal places) there will be a loss in accuracy as we cannot retain all information... one could imagine it like rounding errors. It is also referred to as catastrophic cancellation. See more details here.

In this case, the order in which we add up the numbers will matter! As I mentioned earlier, the parallelisation of operations will mean that we cannot know the order of the operations, and so we cannot guarantee identical answers for the same runs of an algorithm, while still enjoying the parallel computations!

Practicality

Although this might cause a headache for some people, because reproducibility is a big - both in academic research as well as industry applications - the variation in results due to this pseudo-randomness and parallelisation/summation errors is really negligible in the bigger picture.

Changing a layer in a deep NN, altering the learning rate or the regularisation are all factors that are much more important and will make larger differences in results. They also encode knowledge and decisions made by you, as a practitioner. I would suggest spending time thinking about these things and not at all worry about these small blips.


Bonus

There is a nice post from Python Guru & Core DEv: Raymond Hettinger, where he shows how to maintain full precision for summations of floating point numbers. It involves keeping track of sub-totals, which can be used to ensure the final sum did not cause any loss of precision.

$\endgroup$
0
$\begingroup$

The results (loss after a certain number of epochs) will be the same every time if you initialize the pseudorandom number generator (np.random.seed(1234)) before importing Keras and restart the Python interpreter between the runs.


Edit: The above solution works with Theano backend but not with TensorFlow backend.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.