# Reason for Huge Jump in Loss For One Epoch Only?

I was wondering what possible reasons there could be for a huge jump in loss for only one epoch during training. I am getting a result like...

Epoch 1/10
2020-05-13 18:42:19.436235: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:184] Filling up shuffle buffer (this may take a while): 30910 of 40000
2020-05-13 18:42:22.360274: I tensorflow/core/kernels/data/shuffle_dataset_op.cc:233] Shuffle buffer filled.
200/200 [==============================] - 173s 863ms/step - loss: 0.1844 - val_loss: 0.4250
Epoch 2/10
200/200 [==============================] - 167s 833ms/step - loss: 80890.9766 - val_loss: 0.5157
Epoch 3/10
200/200 [==============================] - 166s 830ms/step - loss: 0.0549 - val_loss: 0.2966
Epoch 4/10
200/200 [==============================] - 170s 849ms/step - loss: 0.0488 - val_loss: 0.2708


Which strikes me as very odd. This is a 3-lay LSTM network using Keras.

I graphed the data after normalization, but nothing out of the ordinary appears.

Here is the code I used to create the network...

# create new more complicated network
multi_step_model = tf.keras.models.Sequential()
return_sequences=True,
input_shape=
[past_history,len(features_considered)]))
# output layers are same as prediction count
multi_step_model.compile(optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0), loss='mae')

# run training
multi_step_history = multi_step_model.fit(train_data_multi, epochs=EPOCHS,
steps_per_epoch=EVALUATION_INTERVAL,
validation_data=val_data_multi,
validation_steps=50)

• Does this happen every time you train? Or is this something that happened once? What optimizer are you using? – Valentin Calomme May 13 at 10:20
• It happens on and off in different epochs. It happens often, sometimes in multiple epochs, but sometimes happens never. I am using RMSprop as the optimizer. – raeldor May 16 at 4:11
• I think your question may have lead to a solution, though not yet an answer as to why. The code I noticed said 'optimizer=tf.keras.optimizers.RMSprop(clipvalue=1.0)'. I change this to 'optimizer=tf.keras.optimizers.RMSprop()' and the problem seems to have gone away. I guess I will need to look and see what this parameter is doing. Thank you! – raeldor May 16 at 5:35
• According to the documentation, it clips gradients that have a bigger absolute value. If let's say, for that epoch, most gradients are clipped, this could lead to your model "going in the completely wrong direction", which could explain the spike in the loss. I updated my answer – Valentin Calomme May 16 at 10:49