5
$\begingroup$

I have a Lending club dataset from Kaggle; it contains many different columns: there are for example dummy variables, years, amount of the loan...ect I want to normalize the data in the training and test set but I have to use the Min and Max of the train set to prevent data leakage from the test set. My question is: if there is, in the test set or even when I try to predict new data point, a value that is greater that the Max value or lower than the Min value and I normalize it using the same values from the train set, is it correct? can I the model process this value normally?

this is the code that I use to normalize

    from  sklearn.preprocessing import MinMaxScaler

    scaler = MinMaxScaler()
    X_train = scaler.fit_transform(X_train)
Improve this question
$\endgroup$
4
$\begingroup$

The minimum and maximum values are just known limits that are parts of the formula that reshapes the distribution of the data, so if a value is bigger than the previously known value the resulting feature scaling (Normalization) will be still appropriate.

An alternative is z-scores if you don't feel like using minimum and maximum values.

x'= (x-x̄) / σ Where x is the original feature vector, x̄ is the average of the vector x is the mean of that feature vector and σ is its standard deviation.

Improve this answer
$\endgroup$
2
  • $\begingroup$ But I know that there is a difference between applying z-score and applying MinMaxScaler? $\endgroup$ – Ghassen Ben Hamida Mar 26 '20 at 11:55
  • 1
    $\begingroup$ Yes, there are always differences in standardization methods and results will vary depending on the combination of standardization and ML algorithm and the data characteristics and size. Some ML algorithms will be able to process the data in its raw form while others won't. BoxCox will have slightly different results than min-max normalization and so on. $\endgroup$ – wacax Apr 2 '20 at 16:49
3
$\begingroup$

In Machine Learning, you are making the assumption that the training and test sets follow the same distribution. If this assumption does not stand, then your model won't be able to generalize properly.

Having said that, there obviously is a chance of a test-set feature having a value slightly larger than the max of that same feature in the training set. If this is the case, all ML models will work perfectly fine for that sample having a normalized value slightly higher than $1$.

What I want to emphasize, however, is that if the training set and the test set have significantly different distributions (most commonly due to a small dataset size), then no model will be able to generalize properly and it won't be a problem of normalization.

Improve this answer
$\endgroup$
4
  • $\begingroup$ For years for example i think that they should not be normalized wht do u say about that? $\endgroup$ – Ghassen Ben Hamida Mar 26 '20 at 11:56
  • 1
    $\begingroup$ It doesn't matter what each variable represents, they all should be normalized. The reason is that some algorithms (most notably distance based ones, e.g. kNN) might give variables with a larger range of values more importance than ones with a smaller one. $\endgroup$ – Djib2011 Mar 26 '20 at 13:10
  • $\begingroup$ so the model will be restricted to work with year values between the min and max year of the training set, doesn't he? $\endgroup$ – Ghassen Ben Hamida Mar 26 '20 at 15:15
  • 1
    $\begingroup$ Not necessarily, but close to those. E.g. if your training set is from 1970-1980 and your test set is from 2000-2010 you might have a problem $\endgroup$ – Djib2011 Apr 1 '20 at 7:31
1
$\begingroup$

You should first normalize the whole dataset then split the data. Though you may split first and then do the separate Normalization. Normalization is only needed when you are using a model which is impacted if the features have different stretch in space e.g. Gradient Descent. It is not needed for Decision tree, Random Forest.

Improve this answer
$\endgroup$
1
  • 3
    $\begingroup$ If I normalize all the dataset, there will be a data leakage from the test set. The model should never see the test data because it is used to evaluate the model $\endgroup$ – Ghassen Ben Hamida Mar 29 '20 at 0:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.