1
$\begingroup$

I am using keras and Jupyter notebook and want to make my results reproducible every time I ran it. This is the tutorial I used https://machinelearningmastery.com/time-series-prediction-lstm-recurrent-neural-networks-python-keras/. I copied his codes in Stacked LSTMs with Memory Between Batches part.

This is my cell1 in Jupyter Notebook, I only used CPU to avoid randomness brought by GPU, making sure the same results can be reproduced every time.

import os
os.environ["CUDA_DEVICE_ORDER"] = "PCI_BUS_ID"   # see issue #152
os.environ["CUDA_VISIBLE_DEVICES"] = ""
from tensorflow.python.client import device_lib
device_lib.list_local_devices()

This is cell2, stricting following the suggestions from this question https://stackoverflow.com/questions/32419510/how-to-get-reproducible-results-in-keras

# Seed value
# Apparently you may use different seed values at each stage
seed_value= 0

# 1. Set the `PYTHONHASHSEED` environment variable at a fixed value
import os
os.environ['PYTHONHASHSEED']=str(seed_value)

# 2. Set the `python` built-in pseudo-random generator at a fixed value
import random
random.seed(seed_value)

# 3. Set the `numpy` pseudo-random generator at a fixed value
import numpy as np

np.random.seed(seed_value)

import tensorflow as tf

tf.compat.v1.set_random_seed(seed_value)

tf.random.set_seed(seed_value)

This is cell3 from his codes, (changed it a little for example, from keras to tensorflow.keras)

import numpy
import matplotlib.pyplot as plt
from pandas import read_csv
import math
from tensorflow.keras.models import Sequential, Model
from tensorflow.keras.layers import Dense, LSTM

from sklearn.preprocessing import MinMaxScaler
from sklearn.metrics import mean_squared_error

# convert an array of values into a dataset matrix
def create_dataset(dataset, look_back=1):
    dataX, dataY = [], []
    for i in range(len(dataset)-look_back-1):
        a = dataset[i:(i+look_back), 0]
        dataX.append(a)
        dataY.append(dataset[i + look_back, 0])
    return numpy.array(dataX), numpy.array(dataY)

# load the dataset
dataframe = read_csv('airline-passengers.csv', usecols=[1], engine='python')
dataset = dataframe.values
dataset = dataset.astype('float32')
# normalize the dataset
scaler = MinMaxScaler(feature_range=(0, 1))
dataset = scaler.fit_transform(dataset)
# split into train and test sets
train_size = int(len(dataset) * 0.67)
test_size = len(dataset) - train_size
train, test = dataset[0:train_size,:], dataset[train_size:len(dataset),:]

# reshape into X=t and Y=t+1
look_back = 3
trainX, trainY = create_dataset(train, look_back)
testX, testY = create_dataset(test, look_back)
# reshape input to be [samples, time steps, features]
trainX = numpy.reshape(trainX, (trainX.shape[0], trainX.shape[1], 1))
testX = numpy.reshape(testX, (testX.shape[0], testX.shape[1], 1))

And this is cell4,

# create and fit the LSTM network
batch_size = 1
model = Sequential()
model.add(LSTM(4, batch_input_shape=(batch_size, look_back, 1), stateful=True, return_sequences=True))
model.add(LSTM(4, batch_input_shape=(batch_size, look_back, 1), stateful=True))
model.add(Dense(1))
model.compile(loss='mean_squared_error', optimizer='adam')
for i in range(100):
    model.fit(trainX, trainY, epochs=1, batch_size=batch_size, verbose=2, shuffle=False)
    model.reset_states()
# make predictions
trainPredict = model.predict(trainX, batch_size=batch_size)
model.reset_states()
testPredict = model.predict(testX, batch_size=batch_size)
# invert predictions
trainPredict = scaler.inverse_transform(trainPredict)
trainY = scaler.inverse_transform([trainY])
testPredict = scaler.inverse_transform(testPredict)
testY = scaler.inverse_transform([testY])
# calculate root mean squared error
trainScore = math.sqrt(mean_squared_error(trainY[0], trainPredict[:,0]))
print('Train Score: %.2f RMSE' % (trainScore))
testScore = math.sqrt(mean_squared_error(testY[0], testPredict[:,0]))
print('Test Score: %.2f RMSE' % (testScore))
# shift train predictions for plotting
trainPredictPlot = numpy.empty_like(dataset)
trainPredictPlot[:, :] = numpy.nan
trainPredictPlot[look_back:len(trainPredict)+look_back, :] = trainPredict
# shift test predictions for plotting
testPredictPlot = numpy.empty_like(dataset)
testPredictPlot[:, :] = numpy.nan
testPredictPlot[len(trainPredict)+(look_back*2)+1:len(dataset)-1, :] = testPredict
# plot baseline and predictions
plt.plot(scaler.inverse_transform(dataset))
plt.plot(trainPredictPlot)
plt.plot(testPredictPlot)
plt.show()

However, I still found that the training loss is different every time I ran cell4.

But I found that, as long as I add cell2's contents to cell4, I can get the same training loss curve every time I ran cell4.

So my question is, to reproduce my results, why should I set the random seed every time I run my model in the cell(cell4), instead of just setting it in the beginning of my jupyter notebook once and for all?

$\endgroup$

1 Answer 1

1
$\begingroup$

Did you set the same random seed at each step?

The seed works well for the first function, but then it is lost in the next ones because NumPy applies a global seed reset automatically.

For example, you can do:

def reset_seed(seed_value):
    np.random.RandomState(seed_value)
    tf.compat.v1.set_random_seed(seed_value)
    tf.random.set_seed(seed_value)

for i in range(100):
    reset_seed(seed_value)
    model.fit(trainX, trainY, epochs=1, batch_size=batch_size, verbose=2, shuffle=False)
    model.reset_states()

Otherwise, you can also use a function with a random number generator: https://albertcthomas.github.io/good-practices-random-number-generators/

$\endgroup$
4
  • $\begingroup$ Thank you very much! I have repeated the same results even on GPU. I have created a reset_seed function. I didn't realize I need to reset the seed every time before I fit the model. $\endgroup$
    – user900476
    Jun 29 at 20:46
  • $\begingroup$ You’re welcome. Not sure if the reset is necessary for TF though. $\endgroup$ Jun 29 at 21:32
  • $\begingroup$ > NumPy applies a global seed reset automatically. Would you mind giving me a link about this? I would like to know its details. $\endgroup$
    – user900476
    Jun 30 at 2:50
  • $\begingroup$ About Numpy? The documentation is well explained numpy.org/doc/1.18/reference/random/index.html $\endgroup$ Jun 30 at 6:41

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.