# LSTM RNN regression: validation loss erratic during training

While training my NN, I am getting unfamiliar behaviors which I don't understand. Firstly: highly erratic validation loss while training loss goes down steadily. Secondly: training loss goes down very slowly (not the usual exponential decay curve)

My model architecture is:

# Create model
model = Sequential()

# Compile model
model.compile(loss='mean_absolute_percentage_error', optimizer=opt, metrics=['MeanAbsolutePercentageError'])


Training set: composed of 30k sequences, sequences are 180x1 (single feature), trying to predict the next element of the sequence.

Validation set: same as training but smaller sample size

Loss = MAPE

Batch size = 32

Training looks like this (green validation loss, red training loss):

Example sequences from training set:

From validation set:

Please help explain this to me, and how can I improve training. PS I am fairly new to ML, any advice would be appreciated

• Are there any differences in how you process train and validation data? If batch size for validation is considerably smaller than that of train, you might get huge variances across different batches. It's not restricted to RNNs. Also, you might want to try a non-linear activation function for the hidden fully connected layer. Jun 13 at 8:07
• Thanks @Übermensch, for processing I fitted a sklearn pipeline to training and applied it to validation dataset. Regarding the batch size, the same is applied to training and validation = 32. Which hidden layer the LSTM or Dense layer?
– Rpg
Jun 13 at 12:21
• For the dense layer, does increasing batch size for both changes the graph in any significant way? Jun 13 at 16:58