I have a CNN network in keras. I do the training on a cloud GPU. I get completely different accuracy and loss graphs when I run the training twice. I set random seeds as below, still no luck. Anything missing? Or is this normal behaviour on external GPU? I read that sometimes that they induce randomness because of certain libraries they might be using? I see %2-4 difference in accuracy everytime I run. So makes it difficult to judge my hyperparameter tuning.

import numpy as np
import tensorflow as tf
import keras

from keras.layers import LeakyReLU
from keras.models import Sequential
from keras.layers import Activation
from keras.layers import Convolution2D, MaxPooling2D, Dropout, Reshape
from keras.layers.pooling import GlobalAveragePooling2D
from keras.optimizers import Adamax, Adadelta, Adagrad, Adam, RMSprop
from keras.callbacks import EarlyStopping, ModelCheckpoint, 
from keras.regularizers import l2,l1,l1_l2
from sklearn.metrics import precision_recall_fscore_support, 
from keras.models import model_from_json
from keras.layers.normalization import BatchNormalization
from sklearn.metrics import confusion_matrix, f1_score, 
 precision_score, recall_score
import keras.backend as K
from keras.layers import Dense, Dropout, Flatten
from sklearn.preprocessing import Normalizer

1 Answer 1


The Dropout layer induces randomness (noise) in the training, because random neurons get disabled in every epoch. This leads to slightly different results per training, but the overall performance should be similar, especially for a large amount of epochs.

In order to set the seed for the Droupout layer, you should modify the seed=None argument when you initialize the layer as keras.layers.Dropout(rate, noise_shape=None, seed=None), as explained in the official webpage.

  • $\begingroup$ But shouldn't those neurons be same since I set a random seed? isn't purpose of setting seeds in advacnce is to prevent such scenarios? $\endgroup$
    – Spring
    Jan 27, 2019 at 14:33
  • $\begingroup$ I edited my answer, please check :) $\endgroup$
    – pcko1
    Jan 27, 2019 at 14:39
  • $\begingroup$ thanks. It helped a bit but still not very consistent. Maybe something else too where I need to do this? $\endgroup$
    – Spring
    Jan 27, 2019 at 15:34
  • $\begingroup$ not sure what else you should change. However, you can always build n models and then average their results :) $\endgroup$
    – pcko1
    Jan 27, 2019 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.