1
$\begingroup$

while normalising the data everone is saying that we need to fit only on x_train and not on x_test ? why is that we should not fit x_test ?

if we should not fit the scaler on x_test then why we need to apply the transformation alone on x_test ?

from sklearn.preprocessing import MinMaxScaler
Scaler=MinMaxScaler()
Scaler.fit(X_train)
x_train=Scaler.transform(x_train)
x_test=Scaler.transform(x_test)
| improve this question | | | | |
$\endgroup$
1
$\begingroup$

If you would fit the scaler using x_test you would be using information from your test set and would be leaking data. This is information that you would not have if your model was in production and can therefore not use when fitting your model.

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ ,what is data leak happening here ? so fit should not be done on x_test if that the case then why transform is done then ? will data leak not happen in transform step ? My query is simeple x-train-> fit and transform is done . x_test - fit is not done so transform should also not be done right ? why transform is done of x_test ? can you explain it with a simple example on what happens in the background ? $\endgroup$ – Aj_MLstater Dec 29 '19 at 17:17
1
$\begingroup$

In addition to the comment made by Oxbowerce, you can reason about it as follows: in a real case, you would expect the distribution of your X_train data to be similiar to the X_test data, so applying the MinMaxScaler transformer to the X_test data fitted "only" on X_train means (or should mean) no actual difference compared to fitting it also with X_test; after all, what this scaler does is finding the min and max values of such distribution and rescaling with these values.

Nevertheless, you could refit your transformer from time to time in a production scenario if you get more and more data with all that available new data...

| improve this answer | | | | |
$\endgroup$
  • $\begingroup$ i able to understand only the first 3 lines i did not get what your trying to say after 3rd line " no actual difference compared to fitting it also with X_test; after all, what this scaler does is finding the min and max values of such distribution and rescaling with these values." . My query is simeple x-train-> fit and transform is done . x_test - fit is not done so transform should also not be done right ? why transform is done of x_test ? can you explain it with a simple example on what happens in the background ? $\endgroup$ – Aj_MLstater Dec 29 '19 at 17:21
  • $\begingroup$ I will show in a new answer how it works with an example $\endgroup$ – German C M Dec 29 '19 at 23:44
0
$\begingroup$

The way the scikit-learn MinMaxScaler works is:

  • fit operation: finds the minimum and maximum values of your feature column (mind this scaling is applied separately for each one of your dataframe attributes/columns)
  • transform: applies the min max scaling operation, with the values found in the 'fit' operation

Worked example:

let's assumme we have one feature with the next values: [285, 543, 511, 359, 338, 678, 519, 760, 792, 395, 320, 952, 653, 129, 306, 91, 284, 359, 595, 167, 112, 802, 740, 977, 753, 74, 908, 353, 168, 622, 613, 602, 786, 396, 957, 600, 457, 74, 825, 547, 983, 825, 978, 823, 503, 782, 41, 574, 42, 851, 532, 225, 71, 827, 215, 25, 857, 682, 93, 218, 526, 776, 425, 688, 557, 61, 218, 984, 339, 92, 623, 762, 147, 634, 159, 24, 634, 365, 772, 283, 954, 805, 736, 678, 445, 677, 175, 484, 508, 341, 719, 599, 118, 401, 405, 349, 268, 133, 902, 804]

What we need is to rescale each value by applying the min max scaling definition:

rescaled_X_value = (X_value - feature_values_min) / (feature_values_max - feature_values_min)

The point is, what are our min and max values? This is what the 'fit' operation from the scikit-learn MinMaxScaler does with your train data; the point is that once you use this 'fit' operation on train data to find the min and max values, you do not have to repeat it on the test data, you only have to apply the 'transform' operation to just rescale the test data. Let's see how it works:

If we apply it manually:

min_max_scaled_train_data = (train_feature_data - train_feature_data.min())/(train_feature_data.max()-train_feature_data.min())
min_max_scaled_train_data[:10]

The result on the first 10 elements is:

array([0.27111575, 0.54014599, 0.50677789, 0.34827946, 0.32638165,
       0.68091762, 0.51511992, 0.76642336, 0.79979145, 0.38581856])

And with scikit learn scaler:

from sklearn.preprocessing import MinMaxScaler

min_max_scaler = MinMaxScaler()
min_max_scaler.fit(train_feature_data.reshape(-1, 1))

The min_max_scaler has already the info (i.e. min and max values) to be applied on your new data (let's say your test data), without having the fit again. We can also see that the result is the same as doing it manually as above:

min_max_scaler.transform(train_feature_data.reshape(-1, 1))[:10]
array([[0.27111575],
       [0.54014599],
       [0.50677789],
       [0.34827946],
       [0.32638165],
       [0.68091762],
       [0.51511992],
       [0.76642336],
       [0.79979145],
       [0.38581856]])

As an additional check, you can see that the difference between the train_data and all your data (i.e. train + test) when finding the min and max values is not that big, because both come from the same distribution. You can find the example full code here

| improve this answer | | | | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.