I have a Keras model. which is defined as follows:

nn_model = Sequential()
nn_model.add(Dense(300, activation="relu",input_shape=(4,))) # because we have 4 features
nn_model.add(Dense(150, activation="relu"))
nn_model.add(Dense(50, activation="relu"))
nn_model.add(Dense(30, activation="relu"))

optimizer = optimizers.RMSprop(0.001)      

This model is used as a regressor for predicting a certain pattern. For the SAME training and testing data AND SAME code, the model is making two entirely different predictions on two different hardwares. In one case, the weights become entirely corrupt and makes flat predictions (training data has no flat curve).

Versions of libraries being used are same on both the machines.

Here are the specs of two machines:

Correct predictions machine: Lenovo ThinkPad E570 model type: Intel(R) Core(TM) i7-7500U CPU @ 2.70GHz

Wrong prediction (AWS VM): Machine type: c5.large

Can anyone point out any reason of which this might be happening?

  • $\begingroup$ Possibly a bug on AWS? Is it the same version of the full stack? $\endgroup$ Dec 21 '18 at 11:36
  • $\begingroup$ Did you set your random seed to be the same on both platforms? It's possibly you just converged to a bad solution on one and not the other if they aren't forced to use the same seed. Certainly possible that it's something else too, but using the same seed will show if it could be that. $\endgroup$
    – Engineero
    Dec 21 '18 at 14:46

Consistency in models / predictions in Keras depend on the following:

  • Software Stack: The most uncommon problem, sometimes there are differences in different version of the numerical software which may change results.

  • Difference in Backends: Keras supports the TensorFlow backend, the Theano backend, and the CNTK backend. All of which will yield slightly different models. In the python console you can check the version with a message similar to:

    Using TensorFlow backend.

You can switch backends using the following guide: Backend utilities | Keras

  • Setting seeds: In order to have consistency among models you need to set random seeds accordingly. In the case of Keras and Tensorflow backend, you need to set two random seeds:

In numpy

from numpy.random import seed


In tensorflow

from tensorflow import set_random_seed


There are other more intricate details for setting seeds in certain parts of the graph in Tensorflow explained better on this Stack Overflow answer and the set_random_seed() API documantation on Tensorflow's website.

Intuitively in your problem, the source of randomness comes from not having set the appropriate seeds in numpy and Tensorflow, so start from there and start exploring the other options and documentations. Also remember that if you're using a different backend (Theano or CNTK), setting seeds will require a different function to be called.

An additional recommendation is to look for bugs in the script, version of downloaded binaries, warnings and errors during installation in case the result are too different. Something could be broken if one instance doesn't perform as expected and a fresh install might be necessary.

  • $\begingroup$ Both the instances are using Tensorflow backend. Furthermore, no GPU is being used. I hadn't tried the random seeds method. I have tried that now but still, the problem is not solved. $\endgroup$ Dec 26 '18 at 5:48

The problem was in the CPU credits system of AWS. Apparently, limits of those credits were being hit which were affecting the training of the model. Upon increasing the CPU credits for the AWS instance, the performance of the instance started matching with the one available locally.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.