# Keras Sequential model returns loss 'nan'

I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
while True:
start_range = 0
for batch in batch_count:
end_range = (start_range + batch[1])
batch_dataset = training_dataset[start_range:end_range]
batch_labels = training_dataset_labels[start_range:end_range]
start_range = end_range
yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

keras.layers.Input(
shape = (training_dataset.shape[1], )
)
)
keras.layers.Dense(
units=training_dataset.shape[1] + 10,
input_shape = (training_dataset.shape[1] + 10,),
kernel_initializer='random_uniform',
bias_initializer='zeros',
activation='relu')
)
keras.layers.Dense(
units=1,
input_shape = (1, ),
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
activation='sigmoid')
)

print('Compiling model...\n')

mlp.compile(
loss=listnet_loss
)

mlp.summary() # print model settings

# Training
with tf.device('/GPU:0'):
print('Start training')
#mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)


How can I do?

• have you checked for nan ion your data set ? Feb 19 '20 at 13:24
• For how many epochs did you train and see? Feb 19 '20 at 13:33
• @lcrmorin I’m pretty sure that my dataset doesn’t contain nan elements. However, I notice that the loss turn to nan when I changed training method: I was using only fit and the loss wasn’t nan, now I’m using fit_generator and it’s nan. Feb 19 '20 at 14:01
• @Sharan for 10 epochs. Feb 19 '20 at 14:01
• @Sharan @Icrmorin, another thing that I notice is that with fit_generator()the training go slower compared with use of fit(). The batch size with fit()was 3. Feb 19 '20 at 15:01

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:

• Add regularization to add l1 or l2 penalties to the weights. Otherwise, try a smaller l2 reg. i.e l2(0.001), or remove it if already exists.
• Try a smaller Dropout rate.
• Clip the gradients to prevent their explosion. For instance in Keras you could use clipnorm=1. or clipvalue=1. as parameters for your optimizer.
• Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull().any()
• Replace optimizer with Adam which is easier to handle. Sometimes also replacing sgd with rmsprop would help.
• Use RMSProp with heavy regularization to prevent gradient explosion.
• Verify that you are using the right activation function (e.g. using a softmax instead of sigmoid for multiple class classification).
• Try to increase the batch size (e.g. 32 to 64 or 128) to increase the stability of your optimization.
• Check the size of your last batch which may be different from the batch size.
• This is what I got for first 3 epoches after I replaced relu with tanh (high loss!): Epoch 1/10 1/1 - 9s - loss: 91189.1953 Epoch 2/10 1/1 - 0s - loss: 91176.1953 Epoch 3/10 1/1 - 0s - loss: 91164.1172 ... When I deleted 0s and 1s from my each row, the results got better loss around 0.9. But deleting those values is not a good idea since those values mean off and on of switches. Any idea about that please?
– Avra
Jul 9 at 3:32
• Thank you very much! It works, but should I add such a regulaizer to every layer given I have LSTM autoencoder structure please? I added it to every layer and loss still around 0.9 for my model. I don't know why is that please?
– Avra
Jul 9 at 3:37
• If batch size fixes your problem, you may have a naive normalization function that doesn't account for zero-division if there's 0-variance in a batch. z = (value - mean) / (std + 1E-7) or any other small value should actually fix the root cause, whereas changing the batch size just makes it less likely to occur. +1 for this being the most comprehensive answer out of about a dozen of these questions. So many answers amount to "I changed X to y and it worked!", which, 9/10 times aren't addressing the real problem (which could be any of these and more). Jul 20 at 17:00

A similar problem was reported here: Loss being outputed as nan in keras RNN. In that case, there were exploding gradients due to incorrect normalisation of values.

Make sure that keras.preprocessing.sequence.pad_sequences() does not have the argument value=None but either value=0.0 or some other number that does not occur in your normal data.