2
$\begingroup$

I have the following Keras/TensorFlow code:

my_initializer = keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=1)

my_model = keras.models.Sequential([
    keras.layers.Dense(200, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu",input_shape=[len(features)]),
    keras.layers.Dense(100, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(50, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(25, kernel_initializer=my_initializer, bias_initializer=my_initializer, activation="relu"),
    keras.layers.Dense(1, kernel_initializer=my_initializer, bias_initializer=my_initializer)
])

my_model.compile(loss="mean_squared_error", optimizer="adam")
history = my_model.fit(train, train_y, epochs=70, batch_size=32)

As far as I know, I always run it with the same training data. After training, I print model parameters:

for layer in my_model.layers: print(layer.get_config(), layer.get_weights())

After every training, the parameters are different and the trained model produces different results on the validation data (the difference is 5-10% on each validation example; the overal performance is much more stable).

As far as I know, I do not use dropout (at list I did not enable it explicitly). Initial model parameter values are initialized with a seed (see the code).

Does anyone have any idea what could be wrong? I am relatively new to neural nets.

$\endgroup$
4
  • $\begingroup$ How is your validation data chosen? Is it the same each time or is it chosen randomly? $\endgroup$
    – Ethan Yun
    Jul 28, 2021 at 18:41
  • $\begingroup$ @EthanYun It's always the same. I have 5-10% difference on each example. The overal performance is much more stable. $\endgroup$
    – Alexey
    Jul 28, 2021 at 18:42
  • $\begingroup$ Since this is a regression model, do you have results other than percentage differences? (MSE/RMSE) $\endgroup$
    – Ethan Yun
    Jul 28, 2021 at 18:45
  • $\begingroup$ @EthanYun, I have RMSLE which is relatively stable (there are about 1500 validation examples). But result for each validation example can vary by up to 10%. $\endgroup$
    – Alexey
    Jul 28, 2021 at 18:55

3 Answers 3

4
$\begingroup$

Another source of randomness is the shuffling of training data before each epoch, enabled by default for model.fit() according to the docs.

Try adding this at the beginning of your code block, it should make your results reproducible (see docs):

keras.utils.set_random_seed(1)

(If this is for research or a real-world application, even if you achieve deterministic results you should still report the impact of randomness on instance-level model performance you observed, e.g. by running the code with 10 different random seeds and reporting mean and standard deviation for validation set and single example performance.)

$\endgroup$
1
  • $\begingroup$ Shuffling of training data is a source of randomness that I did not think of when I asked my question. $\endgroup$
    – Alexey
    Jul 18, 2022 at 13:38
1
$\begingroup$

It’s not wrong. NN training is inherently stochastic. As an optimisation problem, the tuning of a NN depends on the initialisation (initialisation of the weights). So the result (the local minimum you end up in) depends on the initialisation too.

There are mainly two ways to go :

  • if this is not a problem for your use case (if only the global performance matter to you) simply set the seed
  • if this is a problem you may consider differents approaches, that all boil down to some sort of regularisation. L1, L2 regularisation, Gaussian noise, drop out, multiple initialisation, ensembling all usually reduce that variance. You’ll probably have to trade off between the stability and the time you are willing to spend hyper parameter tuning your NN.
$\endgroup$
1
  • $\begingroup$ My initial weights are always the same (I have a seed in my_initializer). Is there any other seed that I can define to make the results more stable? $\endgroup$
    – Alexey
    Jul 28, 2021 at 18:32
1
$\begingroup$

There's nothing "wrong" with your model or code. Your model is just trained using a stochastic method. Meaning that your model will converge on the optimal values in a different way each time.

$\endgroup$
4
  • $\begingroup$ The initial weights are always the same. I ensure it with the my_initializer = keras.initializers.RandomNormal(mean=0.0, stddev=0.05, seed=1) line. I checked that the initial weights are always the same. $\endgroup$
    – Alexey
    Jul 28, 2021 at 18:22
  • $\begingroup$ oh shoot, didn't see the seed $\endgroup$
    – Ethan Yun
    Jul 28, 2021 at 18:32
  • $\begingroup$ So, do you think it is normal to have 5-10% difference in model prediction after every training? $\endgroup$
    – Alexey
    Jul 28, 2021 at 18:35
  • $\begingroup$ 5% - 10% is actually a relatively large difference, but since there's nothing wrong the code shown, I would assume that it's related to the data or parts of the code not shown $\endgroup$
    – Ethan Yun
    Jul 28, 2021 at 18:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.