Why such a big difference in number between training error and validation error?

Question

Question

1. Why such a big difference between my 'Train loss' and 'Validation loss' as shown in the picture below? Is it a signal that my codes are wrong and my trained network is wrong as well?

2. Some of my codes are as follows:

DATA_SPLIT_PCT = 0.2

timesteps =  5
n_features =  20

epochs = 100
batch = 32
lr = 0.0001

lstm_autoencoder = Sequential([
    # Encoder
    LSTM(8, activation='relu', input_shape=(timesteps, n_features), return_sequences=True),
    LSTM(4, activation='relu', return_sequences=False),
    RepeatVector(timesteps),

    # Decoder
    LSTM(4, activation='relu', return_sequences=True),
    LSTM(8, activation='relu', return_sequences=True)
    TimeDistributed(Dense(n_features)),
])

adam = optimizers.Adam(lr)
lstm_autoencoder.compile(loss='mse', optimizer=adam)

for stock in stock_list:  # 500 stocks in stock_list
    lstm_autoencoder_history = lstm_autoencoder.fit(X_train_dict[ticker], X_train_dict[ticker], 
                                                    epochs=epochs, 
                                                    batch_size=batch, 
                                                    validation_data=(X_valid_dict[ticker], X_valid_dict[ticker]),
                                                    verbose=False).history

plt.plot(lstm_autoencoder_history['loss'], linewidth=2, label='Train')
plt.plot(lstm_autoencoder_history['val_loss'], linewidth=2, label='Valid')
plt.show()

3. I used the for loop to feed my data into lstm_autoencoder network. In the dictionary variable stock_list, there are 500 stock names such as 'AAPL'.

4. I plotted lstm_autoencoder_history['loss'] and lstm_autoencoder_history['val_loss'] and it is weird because usually validation loss is higher than train loss.

5. I am curious to know why my plot has smaller amount of validation loss. For your information, I used Keras as my deep learning framework. And since I used Keras, I thought this library would handle the different proportion of training set size and validation set size by averaging the errors.

Answer 1

I think this is because your training set is much larger than your validation set. What this means is that for training; losses are being accumulated for comparatively larger number of examples than validation. Hence your training error is larger than your validation error.

I don't think its a problem with your code. If you see closely the trend line has almost same slope for both training loss and validation loss; this suggests that you are okay as far as code is concerned.