0
$\begingroup$

At the moment I perform the following:

estimators = []
estimators.append(('standardize', StandardScaler()))
prepare_data = Pipeline(estimators)

n_splits = 5
tscv = TimeSeriesSplit(n_splits = n_splits)

for train_index, val_index in tscv.split(df_train):
    X_train, X_val = prepare_data.fit_transform(df_train[train_index]), prepare_data.fit_transform(df_train[val_index])

X_test = prepare_data.fit_transform(df_test)

Now I would like to know if this is correct. My concern is that X_train and X_test are transformed separately. While in the first instance I thought this is how it should be I'm about to change my mind as I think I have to use the mean and std of the train set to use within the test set?

$\endgroup$
3
$\begingroup$

The recommended way (see 'Elements of Statistical Learning', chapter 'The Wrong and Right Way to Do Cross-validation') is to calculate the mean and the standard deviation of the values in the training set and then apply them for standardizing both the training and testing sets.

The idea behind this is to prevent data leakage from the testing to the training set because the aim of model validation is to subject the testing data to the same conditions as the data used for the model training.

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Alright, thank you! Can you also add how I do that? :) Or let's say, how do I get the mean and sd of the training set and apply it to another set? $\endgroup$ – Ben Nov 25 '19 at 9:38
2
$\begingroup$

I guess you are using scikit-learn...

What you have to do is to fit the pipeline with X_train and for X_test only tranform.

With the fit method you will compute the mean and std. dev. on the given data (X_train) and with the transform you apply the transformation with these computed values to a given dataset.

The problem is that in scikit-learn, there is no isolated transform method, it is embbeded in the predict method, that eventually applies all transformations and gives the predictions of the last estimator of the PipeLine.

In this post, how to apply only transformations is explained: https://stackoverflow.com/questions/33469633/how-to-transform-items-using-sklearn-pipeline

|improve this answer|||||
$\endgroup$
  • $\begingroup$ Thanks a lot! So does this mean I have to calculate mean and std on my own for the training set and apply these to the test set? I'm just thinking about to add this obvious method to scikit but on the other side I think there is a reason why it is not already?! $\endgroup$ – Ben Nov 26 '19 at 8:54
  • 1
    $\begingroup$ Scikit does it for you: fit method calculates mean and std on whichever the dataset you choose, and transform applies the transofrmation with the computed values by the fit. So, if you call estimator1.fit_transform(x_train) you compute mean and std on x_train_(and store them) and standarize _x_train. When you callestimator1.transform(x_val) you only apply the standarization without re-computing the paramaters. NOTE: transform method does not exist isolated in scikit-learn $\endgroup$ – ignatius Nov 26 '19 at 9:39
  • $\begingroup$ Ah, now I got it. Alright, thank you very much! $\endgroup$ – Ben Nov 26 '19 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.