2
$\begingroup$

Here's the data normalization process of a time series in a paper about stock prediction using LSTM:

  1. Split train and test set based on time (e.g. training set: 2001-2010, test set:2011-2012). This looks fine to me.

  2. Normalize the training set by subtracting the mean and dividing them by the standard deviation of the training set.

  3. Train the data, using 20% of samples as the validation set. Keras's model, by default, uses the last 20% for validation.

So, in the training phase, the model knows a little about the validation set through the mean and standard deviation in step 2.

On one hand, the model is training using some of the future information. On the other hand, no information of the test set is leaked to the model. Is this a kind of look ahead bias?

And what is the best practices in this case?

$\endgroup$
1
$\begingroup$

To block the data leakage from the validation set to the training set in step (2),

  1. We should first split the data to training and validation sets, then
  2. Calculate the mean and standard deviation only using training set, and finally
  3. Use this mean and std to normalize both the training and validation sets.

This way no information is leaked from validation set to training set, although validation set would be almost normalized. This deviation denotes an informative difference between distribution of validation and training sets which later [rightfully] leads to a validation error larger than training error.

We may think of normalizing the sets separately, however this way we may lose a potential discrepancy between the distribution of validation and training sets.

For example, in a 1D time-series of price, if the average price in training set is $1.0\$$, we expect that a validation set with average $2.0\$$ causes problem for a model that is trained on prices around $1.0\$$. However, by normalizing the validation set separately to $1.0\$$, we wrongly bring the validation set to a range that the trained model already expects, which leads to underestimation of validation error.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.