8
$\begingroup$

I'm implementing a neural network with Keras, but the Sequential model returns nan as loss value. I have sigmoid activation function in the output layer to squeeze output between 0 and 1, but maybe doesn't work properly.

This is the code:

def data_generator(batch_count, training_dataset, training_dataset_labels):
  while True:
    start_range = 0
    for batch in batch_count:
      end_range = (start_range + batch[1])
      batch_dataset = training_dataset[start_range:end_range]
      batch_labels = training_dataset_labels[start_range:end_range]
      start_range = end_range
      yield batch_dataset, batch_dataset

mlp = keras.models.Sequential()

# add input layer
mlp.add(
    keras.layers.Input(
        shape = (training_dataset.shape[1], )
    )
)
# add hidden layer
mlp.add(
    keras.layers.Dense(
        units=training_dataset.shape[1] + 10,
        input_shape = (training_dataset.shape[1] + 10,),
        kernel_initializer='random_uniform',
        bias_initializer='zeros',
        activation='relu')
    )
# add output layer
mlp.add(
    keras.layers.Dense(
        units=1,
        input_shape = (1, ),
        kernel_initializer='glorot_uniform',
        bias_initializer='zeros',
        activation='sigmoid')
    )

print('Compiling model...\n')

mlp.compile(
    optimizer='adam',
    loss=listnet_loss
)

mlp.summary() # print model settings


# Training
with tf.device('/GPU:0'):
  print('Start training')
  #mlp.fit(training_dataset, training_dataset_labels, epochs=50, verbose=2, batch_size=3, workers=10)
  mlp.fit_generator(data_generator(groups_id_count, training_dataset, training_dataset_labels),
                    steps_per_epoch=len(training_dataset), epochs=50, verbose=2, workers=10, use_multiprocessing=True)

How can I do?

$\endgroup$
6
  • $\begingroup$ have you checked for nan ion your data set ? $\endgroup$
    – lcrmorin
    Feb 19 '20 at 13:24
  • $\begingroup$ For how many epochs did you train and see? $\endgroup$
    – Sharan
    Feb 19 '20 at 13:33
  • 1
    $\begingroup$ @lcrmorin I’m pretty sure that my dataset doesn’t contain nan elements. However, I notice that the loss turn to nan when I changed training method: I was using only fit and the loss wasn’t nan, now I’m using fit_generator and it’s nan. $\endgroup$
    – pairon
    Feb 19 '20 at 14:01
  • $\begingroup$ @Sharan for 10 epochs. $\endgroup$
    – pairon
    Feb 19 '20 at 14:01
  • $\begingroup$ @Sharan @Icrmorin, another thing that I notice is that with fit_generator()the training go slower compared with use of fit(). The batch size with fit()was 3. $\endgroup$
    – pairon
    Feb 19 '20 at 15:01
12
$\begingroup$

To sum up the different solutions from both stackOverflow and github, which would depend of course on your particular situation:

  • Add regularization to add l1 or l2 penalties to the weights. Otherwise, try a smaller l2 reg. i.e l2(0.001), or remove it if already exists.
  • Try a smaller Dropout rate.
  • Clip the gradients to prevent their explosion. For instance in Keras you could use clipnorm=1. or clipvalue=1. as parameters for your optimizer.
  • Check validity of inputs (no NaNs or sometimes 0s). i.e df.isnull().any()
  • Replace optimizer with Adam which is easier to handle. Sometimes also replacing sgd with rmsprop would help.
  • Use RMSProp with heavy regularization to prevent gradient explosion.
  • Try normalizing your data, or inspect your normalization process for any bad values introduced.
  • Verify that you are using the right activation function (e.g. using a softmax instead of sigmoid for multiple class classification).
  • Try to increase the batch size (e.g. 32 to 64 or 128) to increase the stability of your optimization.
  • Check the size of your last batch which may be different from the batch size.
$\endgroup$
3
  • $\begingroup$ This is what I got for first 3 epoches after I replaced relu with tanh (high loss!): Epoch 1/10 1/1 - 9s - loss: 91189.1953 Epoch 2/10 1/1 - 0s - loss: 91176.1953 Epoch 3/10 1/1 - 0s - loss: 91164.1172 ... When I deleted 0s and 1s from my each row, the results got better loss around 0.9. But deleting those values is not a good idea since those values mean off and on of switches. Any idea about that please? $\endgroup$
    – Avra
    Jul 9 at 3:32
  • $\begingroup$ Thank you very much! It works, but should I add such a regulaizer to every layer given I have LSTM autoencoder structure please? I added it to every layer and loss still around 0.9 for my model. I don't know why is that please? $\endgroup$
    – Avra
    Jul 9 at 3:37
  • 1
    $\begingroup$ If batch size fixes your problem, you may have a naive normalization function that doesn't account for zero-division if there's 0-variance in a batch. z = (value - mean) / (std + 1E-7) or any other small value should actually fix the root cause, whereas changing the batch size just makes it less likely to occur. +1 for this being the most comprehensive answer out of about a dozen of these questions. So many answers amount to "I changed X to y and it worked!", which, 9/10 times aren't addressing the real problem (which could be any of these and more). $\endgroup$ Jul 20 at 17:00
1
$\begingroup$

A similar problem was reported here: Loss being outputed as nan in keras RNN. In that case, there were exploding gradients due to incorrect normalisation of values.

$\endgroup$
0
$\begingroup$

If you found this via Google and use keras.preprocessing.sequence.pad_sequences to pad sequences to train RNNs:

Make sure that keras.preprocessing.sequence.pad_sequences() does not have the argument value=None but either value=0.0 or some other number that does not occur in your normal data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.