1
$\begingroup$

I want to train several CNN architectures with Google Colab (GPU), Keras and Tensorflow. Since the trained models are not reproducible due to GPU support, I would like to train the models several times and determine the mean and the standard deviation of the results. I'm totally unsure if I should at least try to make the models minimal reproducible? For example with the following code at the beginning of the program:

import numpy
import tensorflow as tf
import random as rn
import os

os.environ['PYTHONHASHSEED']='0'
np.random.seed(1)
rn.seed(1)
tf.set_random_seed(1)

from keras import backend as K
if 'tensorflow' == K.backend():
   import tensorflow as tf
   tf.set_random_seed(1)
   from keras.backend.tensorflow_backend import set_session
   config = tf.ConfigProto()
   config.gpu_options.allow_growth = True
   config.gpu_options.visible_device_list = "0"
   set_session(tf.Session(config=config)

I don't know if this makes sense. Would it be better if I did not use seeds at all? What do you think?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Getting slightly different results is natural and should not be a problem. How to minimize the instabilities due to several contributing factors is discussed at length in the linked post below, for Keras using different backends including TensorFlow:

https://machinelearningmastery.com/reproducible-results-neural-networks-keras/

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I know the linked post. My models are not reproducible, so I asked if it makes sense to make the models at least minimally reproducible despite repeated training? In other words, if I train the same model architecture several times to then determine the mean and standard deviation of the results, would it be good if these trained models are minimally reproducible, or should they have been generated without setting any seeds? $\endgroup$
    – Code Now
    Nov 3, 2019 at 14:06
  • $\begingroup$ Okay, I just wanted to make sure it is clear that this result is to be expected. To answer your question, I would go ahead and run every model with a different random state seed to take into account any possible variance within the model. But assuming you also do cross validation, this might take a very long time on Colab. $\endgroup$
    – serali
    Nov 3, 2019 at 14:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.