1
$\begingroup$

I have a CNN network in keras. I do the training on a cloud GPU. I get completely different accuracy and loss graphs when I run the training twice. I set random seeds as below, still no luck. Anything missing? Or is this normal behaviour on external GPU? I read that sometimes that they induce randomness because of certain libraries they might be using? I see %2-4 difference in accuracy everytime I run. So makes it difficult to judge my hyperparameter tuning.

import numpy as np
np.random.seed(3)
import tensorflow as tf
tf.set_random_seed(4)
import keras
keras.backend.clear_session()


from keras.layers import LeakyReLU
from keras.models import Sequential
from keras.layers import Activation
from keras.layers import Convolution2D, MaxPooling2D, Dropout, Reshape
from keras.layers.pooling import GlobalAveragePooling2D
from keras.optimizers import Adamax, Adadelta, Adagrad, Adam, RMSprop
from keras.callbacks import EarlyStopping, ModelCheckpoint, 
 ReduceLROnPlateau
from keras.regularizers import l2,l1,l1_l2
from sklearn.metrics import precision_recall_fscore_support, 
 roc_auc_score
from keras.models import model_from_json
from keras.layers.normalization import BatchNormalization
from sklearn.metrics import confusion_matrix, f1_score, 
 precision_score, recall_score
import keras.backend as K
from keras.layers import Dense, Dropout, Flatten
from sklearn.preprocessing import Normalizer
$\endgroup$
2
$\begingroup$

The Dropout layer induces randomness (noise) in the training, because random neurons get disabled in every epoch. This leads to slightly different results per training, but the overall performance should be similar, especially for a large amount of epochs.

In order to set the seed for the Droupout layer, you should modify the seed=None argument when you initialize the layer as keras.layers.Dropout(rate, noise_shape=None, seed=None), as explained in the official webpage.

| improve this answer | |
$\endgroup$
  • $\begingroup$ But shouldn't those neurons be same since I set a random seed? isn't purpose of setting seeds in advacnce is to prevent such scenarios? $\endgroup$ – Spring Jan 27 '19 at 14:33
  • $\begingroup$ I edited my answer, please check :) $\endgroup$ – pcko1 Jan 27 '19 at 14:39
  • $\begingroup$ thanks. It helped a bit but still not very consistent. Maybe something else too where I need to do this? $\endgroup$ – Spring Jan 27 '19 at 15:34
  • $\begingroup$ not sure what else you should change. However, you can always build n models and then average their results :) $\endgroup$ – pcko1 Jan 27 '19 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.