3
$\begingroup$

I know that for MinMaxScaler we should apply it to train data, then apply it, with the obtained parameters, over test data:

train_X = scaler.fit_transform( train_X  )
test_X = scaler.transform( test_X )

Now, When

scaler = MinMaxScaler,

the data will be transformed to [0,1]. My question is that: when we have

minimum(test_X)<minimum(train_X)

or

maximum(test_X)>maximum(train_X)

for some features, the the scaled test_X will be out of interval [0,1]. Is this true?

$\endgroup$

1 Answer 1

7
$\begingroup$

From documentation: MinMaxScaler

X_std = (X - X.min(axis=0)) / (X.max(axis=0) - X.min(axis=0))

X_scaled = X_std * (max - min) + min

 feature_rangetuple:  (min, max), default=(0, 1)
      Desired range of transformed data.

 clipbool, default=False
       Set to True to clip transformed values of held-out data to provided feature range.

That is, MinMaxScaler is trained on 1 sample with data, but if the distribution changes, then the transformation results will change. For example, if you were displaying continuous values [0; 5] in [0; 1], and then try to supply the values [0; 10], then your result will be within [0; 2]. To force clipping from 0 to 1, set clip=True.

$\endgroup$
3
  • $\begingroup$ Yes you are right. But is this true? As I checked, based on the clip=True, the output of transformation for data which are less than or equal the minimum is zero, for data greater than or equal the maximum is 1 and other data will be converted based on the formula. Is it not better to normalize all data before splitting to test and train? Also what we should do in k-fold cross validation? $\endgroup$
    – John mx
    Commented Dec 13, 2022 at 7:59
  • 1
    $\begingroup$ I would not do any modeling process, including preprocessing, on the entire sample, rather only on the train dataset so that there are no data leaks. It's normal for data to vary slightly per train/test or cv , especially as data drift is possible in the future. Don't use clipbool = true and continue building the model as you originally intended. This data normalization is needed so that the algorithm converges faster due to the same order of values. Only if there are no hard limits display numbers from 0 to 1. $\endgroup$
    – Andrew
    Commented Dec 13, 2022 at 8:16
  • $\begingroup$ If we transfer training data to [0,1] using max and min of train data, then use this max and min of training data to transfer test data to [0,1], then, the transformed test data may be out of [0,1]. This can be occurred specifically when the max and min of data be greater or less than that of in training data respectively. $\endgroup$
    – John mx
    Commented Dec 31, 2022 at 6:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.