I was wondering what possible reasons there could be for a huge jump in loss for only one epoch during training. I am getting a result like...

Epoch 1/10
2020-05-13 18:42:19.436235: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 30910 of 40000
2020-05-13 18:42:22.360274: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
200/200 [==============================] - 173s 863ms/step - loss: 0.1844 - val_loss: 0.4250
Epoch 2/10
200/200 [==============================] - 167s 833ms/step - loss: 80890.9766 - val_loss: 0.5157
Epoch 3/10
200/200 [==============================] - 166s 830ms/step - loss: 0.0549 - val_loss: 0.2966
Epoch 4/10
200/200 [==============================] - 170s 849ms/step - loss: 0.0488 - val_loss: 0.2708

Which strikes me as very odd. This is a 3-lay LSTM network using Keras.

I graphed the data after normalization, but nothing out of the ordinary appears.

Graph of dataset after normalization

Here is the code I used to create the network...

# create new more complicated network
multi_step_model = tf.keras.models.Sequential()
multi_step_model.add(tf.keras.layers.LSTM(16, activation='relu'))
# output layers are same as prediction count
multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

# run training
multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
  • $\begingroup$ Does this happen every time you train? Or is this something that happened once? What optimizer are you using? $\endgroup$ – Valentin Calomme May 13 at 10:20
  • $\begingroup$ It happens on and off in different epochs. It happens often, sometimes in multiple epochs, but sometimes happens never. I am using RMSprop as the optimizer. $\endgroup$ – raeldor May 16 at 4:11
  • $\begingroup$ I think your question may have lead to a solution, though not yet an answer as to why. The code I noticed said 'optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0)'. I change this to 'optimizer=tf.keras.optimizers.RMSprop()' and the problem seems to have gone away. I guess I will need to look and see what this parameter is doing. Thank you! $\endgroup$ – raeldor May 16 at 5:35
  • 1
    $\begingroup$ According to the documentation, it clips gradients that have a bigger absolute value. If let's say, for that epoch, most gradients are clipped, this could lead to your model "going in the completely wrong direction", which could explain the spike in the loss. I updated my answer $\endgroup$ – Valentin Calomme May 16 at 10:49

I cannot say for sure, but it seems that it might be an error in the printout. The validation loss doesn't seem to spike, so it seems that the loss may not actually be what is printed.

I can only advise to run training multiple times and see if this happens again. If yes, try toying with the learning rate. There is a small chance that the learning rate could be too high making the loss diverge for one epoch.

Based on your code, it seems that your optimizer clips gradient with an absolute value larger than 1. In a scenario where most of your gradients are larger, this could lead to your model being optimized in the "wrong direction". This could lead to the loss being so large for some epochs.

| improve this answer | |
  • $\begingroup$ I thought maybe it was the numeric display too, but the graph also has wild spikes. I tried a few times; sometimes it doesn't come up with the wild spikes, but sometimes it goes crazy. Could a data issue cause this even though I've normalized the data? I printed some test frames and the data all seems to be between -3 and +3 give or take. $\endgroup$ – raeldor May 13 at 12:29
  • $\begingroup$ Could it somehow be that the loss function deals with some divisions by "near-zero" which would cause the loss to spike? Must be some numerical stability issue. Are you using mini-batches? Maybe investigate which batch is causing this $\endgroup$ – Valentin Calomme May 13 at 15:08
  • $\begingroup$ Could be an issue with certain batches of data. That would explain why some runs have more spikes than others. I am using a batch size of 256... not entirely sure what you mean by 'mini' batch. $\endgroup$ – raeldor May 14 at 14:04
  • $\begingroup$ I graphed the data to see if there was anything weird, but it looks fine, so I don't think it's a data quality issue. I added the graph and the network creation code to the post. $\endgroup$ – raeldor May 16 at 4:03

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.