LSTM Validation MSE always lower than Train MSE

Question

I am trying to train a LSTM network to forecast time steps further. I have a list of queries and the current question is based on one among them.

The validation loss (using mse) is always lower than the Train loss (mse), I know I am under fitting hence, generalizing pretty badly.

So atleast overfit for sometime, what changes can be brought in the network. Below is the code for the same.

learning_rate = 0.001 n_neurons = [150, 80, 60, 40, 25, 10] dropout = 0.2

def fit_lstm(train, n_lag, n_seq, n_batch, nb_epoch, n_neurons, dropout=dropout, reset_state=False): # reshape training into [samples, timesteps, features]... X, y = train[:, :n_lag], train[:, n_lag:] X = X.reshape(X.shape[0], 1, X.shape[1]) # design network.. model = Sequential() model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), dropout_U=dropout, stateful=True, return_sequences=True)) model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), dropout_U=dropout, stateful=True, return_sequences=True)) model.add(LSTM(n_neurons[5], batch_input_shape=(n_batch, X.shape[1], X.shape[2]), stateful=True, return_sequences=False)) model.add(Dense(y.shape[1], activation='tanh')) tic = time.time() model.compile(optimizer=adam, loss='mse') #, metrics=[mean_absolute_percentage_error] # fit network.. loss, val_loss = list(), list() for i in range(nb_epoch): print('Running Epoch ==> %s' %i)
history = model.fit(X, y, nb_epoch=1, batch_size=n_batch, validation_split=0.1, callbacks=[early_stop,reduce_lr], verbose=2, shuffle=False) loss.append(history.history['loss']) val_loss.append(history.history['val_loss']) model.reset_states() ## clears the state... toc = time.time() print('====='*10) print('Total computation time to train the Model : %0.2f' %((toc - tic) * 100) + ' secs')
return model, loss, val_loss

Any help would be highly appreciable.

Thanks in advance.

Answer 1

One important thing to start with is to check that your targets are in [-1, 1] range because you have a 'tanh' as output function. You should also analyze the distribution of your targets (True class).

Then, one of the first steps that I would recommend is to try to overfit your model. Make a model 'complex enough' and overfit small proportion of your initial data set (20% maybe?).

If your model does not overfit, there are two possibilities:

1. Model architecture not suited for this task (maybe lstm) or model not complex enough
2. Your data is not adapted for prediction task.

If your model overfits (Good sign):

1. Try to increase the complexity of your model when increasing the size of the samples (training set). Do not use any dropout or regularization and increase the number of layers instead of the number of units/layer.