# Keras Sequential model returns loss 'nan'

I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
while True:
start_range = 0
for batch in batch_count:
end_range = (start_range + batch[1])
batch_dataset = training_dataset[start_range:end_range]
batch_labels = training_dataset_labels[start_range:end_range]
start_range = end_range
yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

keras.layers.Input(
shape = (training_dataset.shape[1], )
)
)
keras.layers.Dense(
units=training_dataset.shape[1] + 10,
input_shape = (training_dataset.shape[1] + 10,),
kernel_initializer='random_uniform',
bias_initializer='zeros',
activation='relu')
)
keras.layers.Dense(
units=1,
input_shape = (1, ),
kernel_initializer='glorot_uniform',
bias_initializer='zeros',
activation='sigmoid')
)

print('Compiling model...\n')

mlp.compile(
loss=listnet_loss
)

mlp.summary() # print model settings

# Training
with tf.device('/GPU:0'):
print('Start training')
#mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)


How can I do?

• have you checked for nan ion your data set ? – lcrmorin Feb 19 at 13:24
• For how many epochs did you train and see? – Sharan Feb 19 at 13:33
• @lcrmorin I’m pretty sure that my dataset doesn’t contain nan elements. However, I notice that the loss turn to nan when I changed training method: I was using only fit and the loss wasn’t nan, now I’m using fit_generator and it’s nan. – pairon Feb 19 at 14:01
• @Sharan for 10 epochs. – pairon Feb 19 at 14:01
• @Sharan @Icrmorin, another thing that I notice is that with fit_generator()the training go slower compared with use of fit(). The batch size with fit()was 3. – pairon Feb 19 at 15:01

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:

• Add regularization to add l1 or l2 penalties to the weights. Otherwise, try a smaller l2 reg. i.e l2(0.001), or remove it if already exists.
• Try a smaller Dropout rate.
• Clip the gradients to prevent their explosion. For instance in Keras you could use clipnorm=1. or clipvalue=1. as parameters for your optimizer.
• Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull().any()
• Replace optimizer with Adam which is easier to handle. Sometimes also replacing sgd with rmsprop would help.
• Use RMSProp with heavy regularization to prevent gradient explosion.