15
$\begingroup$

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before splitting the data into train/test, but when i was checking some of the codes posted online (using sklearn) there were two major uses.

1- Using StandardScaler on all the data. E.g.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_fit = sc.fit(X)
X_std = X_fit.transform(X)

Or

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit(X)
X = sc.transform(X)

Or simply

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_std = sc.fit_transform(X)

2- Using StandardScaler on split data.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

I would like to standardize my data, but I am confused which method is best!

$\endgroup$
19
$\begingroup$

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to transform the training set, which in turn is passed downstream.

As an example, knowing the distribution of the whole dataset might influence how you detect and process outliers, as well as how you parameterise your model. Although the data itself is not exposed, information about the distribution of the data is. As a result, your test set performance is not a true estimate of performance on unseen data. Some further discussion you might find useful is on Cross Validated.

$\endgroup$
  • $\begingroup$ Would the same apply to a validation set? In other words, if I split my Training Set into Train and Validation sets, do I learn the fit on just the train and then apply to both the Validation and Test sets later? or do I learn the fit on the entire data set that comprised both the Validation and Training examples and only worry about applying it to the Test set later. $\endgroup$ – Phil Glau Sep 7 at 6:45
  • $\begingroup$ hi - that's correct, fit only on train - not validation or test $\endgroup$ – redhqs Sep 9 at 14:08
0
$\begingroup$

How about the following:

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.fit_transform (X_test)

Because if X_test = sc.transform(X_test), it returns error X_test is not fitted yet. Or did I miss something here?

$\endgroup$
  • 2
    $\begingroup$ Just remove fit from the X_test $\endgroup$ – tsumaranaina Jun 7 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.