Different results after each training of Keras/TensorFlow model

Question

I have the following Keras/TensorFlow code:

my_initializer = keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=1)

my_model = keras.models.Sequential([
    keras.layers.Dense(200, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu",input_shape=[len(features)]),
    keras.layers.Dense(100, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(50, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(25, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(1, kernel_initializer=my_initializer, bias_initializer=my_initializer)
])

my_model.compile(loss="mean_squared_error", optimizer="adam")
history = my_model.fit(train, train_y, epochs=70, batch_size=32)

As far as I know, I always run it with the same training data. After training, I print model parameters:

for layer in my_model.layers: print(layer.get_config(), layer.get_weights())

After every training, the parameters are different and the trained model produces different results on the validation data (the difference is 5-10% on each validation example; the overal performance is much more stable).

As far as I know, I do not use dropout (at list I did not enable it explicitly). Initial model parameter values are initialized with a seed (see the code).

Does anyone have any idea what could be wrong? I am relatively new to neural nets.

Answer 1

Another source of randomness is the shuffling of training data before each epoch, enabled by default for model.fit() according to the docs.

Try adding this at the beginning of your code block, it should make your results reproducible (see docs):

keras.utils.set_random_seed(1)

(If this is for research or a real-world application, even if you achieve deterministic results you should still report the impact of randomness on instance-level model performance you observed, e.g. by running the code with 10 different random seeds and reporting mean and standard deviation for validation set and single example performance.)

Answer 2

It’s not wrong. NN training is inherently stochastic. As an optimisation problem, the tuning of a NN depends on the initialisation (initialisation of the weights). So the result (the local minimum you end up in) depends on the initialisation too.

There are mainly two ways to go :

• if this is not a problem for your use case (if only the global performance matter to you) simply set the seed
• if this is a problem you may consider differents approaches, that all boil down to some sort of regularisation. L1, L2 regularisation, Gaussian noise, drop out, multiple initialisation, ensembling all usually reduce that variance. You’ll probably have to trade off between the stability and the time you are willing to spend hyper parameter tuning your NN.

Answer 3

There's nothing "wrong" with your model or code. Your model is just trained using a stochastic method. Meaning that your model will converge on the optimal values in a different way each time.