48
$\begingroup$

When I was reading about using StandardScaler, most of the recommendations were saying that you should use StandardScaler before splitting the data into train/test, but when i was checking some of the codes posted online (using sklearn) there were two major uses.

Case 1: Using StandardScaler on all the data. E.g.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_fit = sc.fit(X)
X_std = X_fit.transform(X)

Or

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X = sc.fit(X)
X = sc.transform(X)

Or simply

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_std = sc.fit_transform(X)

Case 2: Using StandardScaler on split data.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)

I would like to standardize my data, but I am confused which approach is the best!

Improve this question
$\endgroup$
1

3 Answers 3

Reset to default
45
$\begingroup$

In the interest of preventing information about the distribution of the test set leaking into your model, you should go for option #2 and fit the scaler on your training data only, then standardise both training and test sets with that scaler. By fitting the scaler on the full dataset prior to splitting (option #1), information about the test set is used to transform the training set, which in turn is passed downstream.

As an example, knowing the distribution of the whole dataset might influence how you detect and process outliers, as well as how you parameterise your model. Although the data itself is not exposed, information about the distribution of the data is. As a result, your test set performance is not a true estimate of performance on unseen data. Some further discussion you might find useful is on Cross Validated.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Would the same apply to a validation set? In other words, if I split my Training Set into Train and Validation sets, do I learn the fit on just the train and then apply to both the Validation and Test sets later? or do I learn the fit on the entire data set that comprised both the Validation and Training examples and only worry about applying it to the Test set later. $\endgroup$
    – Phil Glau
    Sep 7, 2019 at 6:45
  • $\begingroup$ hi - that's correct, fit only on train - not validation or test $\endgroup$
    – redhqs
    Sep 9, 2019 at 14:08
6
$\begingroup$

You shouldn't be doing fit_transform(X_test) on the test data.
The fit already occurred above.

from sklearn.preprocessing import StandardScaler
sc = StandardScaler()
X_train = sc.fit_transform(X_train)
X_test = sc.transform (X_test)
Improve this answer
$\endgroup$
1
  • $\begingroup$ Sorry, this was meant as a reply to what's below :) $\endgroup$
    – starsini
    Oct 10, 2019 at 6:25
0
$\begingroup$

How about the following:

from sklearn.preprocessing import StandardScaler

sc = StandardScaler()

X_train = sc.fit_transform(X_train)

X_test = sc.fit_transform (X_test)

Because if X_test = sc.transform(X_test), it returns error X_test is not fitted yet. Or did I miss something here?

Improve this answer
$\endgroup$
1
  • 6
    $\begingroup$ Just remove fit from the X_test $\endgroup$
    – tsumaranaina
    Jun 7, 2019 at 0:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.