While training my NN, I am getting the following behavior:

  • highly erratic validation loss while training loss goes down steadily.
  • training loss goes down very slowly (not the usual exponential decay curve)

My model architecture is:

# Create model
model = Sequential()
model.add(Bidirectional(LSTM(30, input_shape=(train_x.shape[1:]), return_sequences=True)))
model.add(Dense(4, activation="linear"))
model.add(Dense(1, activation="linear"))
opt = tf.keras.optimizers.Adam()

# Compile model
model.compile(loss='mean_absolute_percentage_error', optimizer=opt, metrics=['MeanAbsolutePercentageError'])

Training set: composed of 30k sequences, sequences are 180x1 (single feature), trying to predict the next element of the sequence.

Validation set: same as training but smaller sample size

Loss = MAPE

Batch size = 32

Training looks like this (green validation loss, red training loss):

enter image description here

Example sequences from training set:

enter image description here enter image description here

From validation set:

enter image description here enter image description here

What is going on and how can I improve training?

  • $\begingroup$ Are there any differences in how you process train and validation data? If batch size for validation is considerably smaller than that of train, you might get huge variances across different batches. It's not restricted to RNNs. Also, you might want to try a non-linear activation function for the hidden fully connected layer. $\endgroup$ Commented Jun 13, 2022 at 8:07
  • $\begingroup$ Thanks @Übermensch, for processing I fitted a sklearn pipeline to training and applied it to validation dataset. Regarding the batch size, the same is applied to training and validation = 32. Which hidden layer the LSTM or Dense layer? $\endgroup$
    – Rpg
    Commented Jun 13, 2022 at 12:21
  • $\begingroup$ For the dense layer, does increasing batch size for both changes the graph in any significant way? $\endgroup$ Commented Jun 13, 2022 at 16:58

1 Answer 1


Try using a 'relu' activation function, it might help. Other than that, increase the batch size to check if the loss descent is smoother or not. Even though the graph is erratic, the overall trend of the loss is going down. Try to overfit the data as much as you can (increase LSTM / Dense layers, epochs), to see if the data is actually learnable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.