Why not use Scaler.fit_transform on total dataframe?

In sklearn I'm normalizing the data with MinMaxScaler. The example I'm following uses

from sklearn.preprocessing import MinMaxScaler
scaler = MinMaxScaler()

X_train, X_test, y_train, y_test = train_test_split(X_crime, y_crime,random_state = 0)

X_train_scaled = scaler.fit_transform(X_train)
X_test_scaled = scaler.transform(X_test)


Now I wonder why this is done separately on the train and test set, and not on the X_crime dataframe like:

X_crime_scaled = scaler.fit_transform(X_crime)

X_train_scaled, X_test_scaled, y_train, y_test = train_test_split(X_crime_scaled, y_crime, random_state = 0)


the R-squared score is higher and with this option I know all my values are normalized between 0 and 1.

If you would use the scaler on the full dataset you would provide the algorithm with some information about the values in the test set that it would not have otherwise.

This additional information ("data leakage") from your test set to your training set is problematic because it leads to unreliable performance estimates. It is therefore not surprising that you achieve a higher R-squared. Due to the data leakage, this R-squared might be overly optimistic because it could depend on the additional information that you introduced into the training set.

This is particularly true for the MinMaxScaler because it is by definition very sensitive to outliers. The effect will probably less problematic if you use the RobustScaler (or even the StandardScaler).

The same holds for other preprocessing steps like outlier removal, feature selection etc.

If you are worried that your training data does not adequately reflect the true distribution, you can fall back to a cross-validation approach with multiple folds so you can estimate the effect across multiple splits of the data. Again, remember to fit the scaler on all the training folds and apply it to the test fold for every iteration.

The main reason for that is the assumption that we make that our model has never seen our test data or have any information about it. So if you run your scaler on the whole dataset, they are probably going to have different max and mins, can you see that? Then, our model will indeed have information about our test data.

If you have further doubts, comment, I'm available to help you again.

• Thanks for your answer, I find it hard to rationalize that normalization adds information to the training set, while I see that the min and max values might be different. Maybe I should give it some time, to sink in. May 5 '18 at 8:03
• Yep, is a harder concept to grasp but these are challenges that you will face everyday. There are some transformations that won't cause your model to know more about the dataset (mostly element/row-wise ones), so keep that in mind, if you see something that is column-wise, get alert right on time. May 5 '18 at 15:40
• Thanks for your reply, as I'm progressing with my tutorial, polynomial feature are introduced and here my lazy way of train_test_split is used so polynomial features do not add lead to data leakage? I think this is a columnwise operation? May 5 '18 at 20:24