I have a CNN network in keras. I do the training on a cloud GPU. I get completely different accuracy and loss graphs when I run the training twice. I set random seeds as below, still no luck. Anything missing? Or is this normal behaviour on external GPU? I read that sometimes that they induce randomness because of certain libraries they might be using? I see %2-4 difference in accuracy everytime I run. So makes it difficult to judge my hyperparameter tuning.
import numpy as np np.random.seed(3) import tensorflow as tf tf.set_random_seed(4) import keras keras.backend.clear_session() from keras.layers import LeakyReLU from keras.models import Sequential from keras.layers import Activation from keras.layers import Convolution2D, MaxPooling2D, Dropout, Reshape from keras.layers.pooling import GlobalAveragePooling2D from keras.optimizers import Adamax, Adadelta, Adagrad, Adam, RMSprop from keras.callbacks import EarlyStopping, ModelCheckpoint, ReduceLROnPlateau from keras.regularizers import l2,l1,l1_l2 from sklearn.metrics import precision_recall_fscore_support, roc_auc_score from keras.models import model_from_json from keras.layers.normalization import BatchNormalization from sklearn.metrics import confusion_matrix, f1_score, precision_score, recall_score import keras.backend as K from keras.layers import Dense, Dropout, Flatten from sklearn.preprocessing import Normalizer