I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
  while True:
    start_range = 0
    for batch in batch_count:
      end_range = (start_range + batch[1])
      batch_dataset = training_dataset[start_range:end_range]
      batch_labels = training_dataset_labels[start_range:end_range]
      start_range = end_range
      yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

# add input layer
        shape = (training_dataset.shape[1], )
# add hidden layer
        units=training_dataset.shape[1] + 10,
        input_shape = (training_dataset.shape[1] + 10,),
# add output layer
        input_shape = (1, ),

print('Compiling model...\n')


mlp.summary() # print model settings

# Training
with tf.device('/GPU:0'):
  print('Start training')
  #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
  mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
                    steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)

How can I do?

  • $\begingroup$ have you checked for nan ion your data set ? $\endgroup$ – lcrmorin Feb 19 '20 at 13:24
  • $\begingroup$ For how many epochs did you train and see? $\endgroup$ – Sharan Feb 19 '20 at 13:33
  • $\begingroup$ @lcrmorin I’m pretty sure that my dataset doesn’t contain nan elements. However, I notice that the loss turn to nan when I changed training method: I was using only fit and the loss wasn’t nan, now I’m using fit_generator and it’s nan. $\endgroup$ – pairon Feb 19 '20 at 14:01
  • $\begingroup$ @Sharan for 10 epochs. $\endgroup$ – pairon Feb 19 '20 at 14:01
  • $\begingroup$ @Sharan @Icrmorin, another thing that I notice is that with fit_generator()the training go slower compared with use of fit(). The batch size with fit()was 3. $\endgroup$ – pairon Feb 19 '20 at 15:01

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:

  • Add regularization to add l1 or l2 penalties to the weights. Otherwise, try a smaller l2 reg. i.e l2(0.001), or remove it if already exists.
  • Try a smaller Dropout rate.
  • Clip the gradients to prevent their explosion. For instance in Keras you could use clipnorm=1. or clipvalue=1. as parameters for your optimizer.
  • Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull().any()
  • Replace optimizer with Adam which is easier to handle. Sometimes also replacing sgd with rmsprop would help.
  • Use RMSProp with heavy regularization to prevent gradient explosion.
  • Try normalizing your data, or inspect your normalization process for any bad values introduced.
  • Verify that you are using the right activation function (e.g. using a softmax instead of sigmoid for multiple class classification).
  • Try to increase the batch size (e.g. 32 to 64 or 128) to increase the stability of your optimization.
  • Check the size of your last batch which may be different from the batch size.

A similar problem was reported here: Loss being outputed as nan in keras RNN. In that case, there were exploding gradients due to incorrect normalisation of values.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.