I want to train several CNN architectures with Google Colab (GPU), Keras and Tensorflow. Since the trained models are not reproducible due to GPU support, I would like to train the models several times and determine the mean and the standard deviation of the results. I'm totally unsure if I should at least try to make the models minimal reproducible? For example with the following code at the beginning of the program:
import numpy import tensorflow as tf import random as rn import os os.environ['PYTHONHASHSEED']='0' np.random.seed(1) rn.seed(1) tf.set_random_seed(1) from keras import backend as K if 'tensorflow' == K.backend(): import tensorflow as tf tf.set_random_seed(1) from keras.backend.tensorflow_backend import set_session config = tf.ConfigProto() config.gpu_options.allow_growth = True config.gpu_options.visible_device_list = "0" set_session(tf.Session(config=config)
I don't know if this makes sense. Would it be better if I did not use seeds at all? What do you think?