40
$\begingroup$

Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

from sklearn.preprocessing import StandardScaler

normalized_X_features = pd.DataFrame(
    StandardScaler().fit_transform(X_features),
    columns = X_features.columns
)

x_train, x_test, y_train, y_test = train_test_split(
    normalized_X_features,
    Y_feature,
    test_size=0.20,
    random_state=4
)
LR = LogisticRegression(
    C=0.01,
    solver='liblinear'
).fit(x_train, y_train)

y_test_pred = LR.predict(x_test)

Normalization after split

x_train, x_test, y_train, y_test = train_test_split(
    X_features,
    Y_feature,
    test_size=0.20,
    random_state=4
)
normalized_x_train = pd.DataFrame(
    StandardScaler().fit_transform(x_train),
    columns = x_train.columns
)
LR = LogisticRegression(
    C=0.01,
    solver='liblinear'
).fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(
    StandardScaler().fit_transform(x_test),
    columns = x_test.columns
)
y_test_pred = LR.predict(normalized_x_test)

So far I have seen both approaches.

$\endgroup$
2
  • 3
    $\begingroup$ The 2nd approach is definitely wrong. You should not be calling a sklearn "fit" method on test data. 'fit' and 'fit_transform' are for training, 'predict' and 'transform' for testing. $\endgroup$ Jul 3, 2019 at 3:49
  • $\begingroup$ For sure after split what techniques you are going to apply for training set,apply the same on test set as well. $\endgroup$ Jan 5, 2022 at 11:44

5 Answers 5

51
$\begingroup$

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.

[Precision thanks to Neil's comment] When normalizing the test set, one should apply the normalization parameters previously obtained from the training set as-is. Do not recalculate them on the test set, because they would be inconsistent with the model and this would produce wrong predictions.

$\endgroup$
7
  • 2
    $\begingroup$ And the test set must also be normalized (before prediction)? $\endgroup$
    – Tauno
    Jul 2, 2019 at 13:22
  • 3
    $\begingroup$ @TaunoTanilas you must treat the test set the same way you treat the training set. $\endgroup$ Jul 2, 2019 at 13:30
  • 3
    $\begingroup$ Yes, test set needs to be normalized before feeding the model. $\endgroup$ Jul 2, 2019 at 13:43
  • $\begingroup$ Important - and I suggest adding to this answer - whatever factors, offsets etc that you derive from the training set to normalise it should be applied as-is to normalise all data used from then on. Do not separately calculate mean, sd etc from the test set to normalise it starting from scratch. Re-use those values established from the training set. $\endgroup$ Jun 30, 2021 at 20:31
  • 1
    $\begingroup$ @StackOverflow no, it's not always needed to normalize anything actually, it depends on the specifics of the task,data and algorithm. You should ask a new question if you want to know about a particular case. $\endgroup$
    – Erwan
    Apr 7, 2022 at 12:26
14
$\begingroup$

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like:

from sklearn.preprocessing import StandardScaler

x_train, x_test, y_train, y_test = train_test_split(
    X_features, 
    Y_feature, 
    test_size=0.20,
    random_state=4)

scaler = StandardScaler()

normalized_x_train = pd.DataFrame(
    scaler.fit_transform(x_train),
    columns = x_train.columns
)

LR = LogisticRegression(
    C=0.01,
    solver='liblinear'
).fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(
    scaler.transform(x_test),
    columns = x_test.columns
)
y_test_pred = LR.predict(normalized_x_test)
$\endgroup$
1
  • 1
    $\begingroup$ Worth calling out that scaler.fit_transform is used on the training set, then scaler.transform is used on the test set as OP gets this wrong in the question. In addition, if this model will be re-used separately to the train, test run then the scaler's fitted params should be stored for re-use (I suppose you could store the training set and re-use it recalculate, but that's quite heavyweight for production use) $\endgroup$ Jun 30, 2021 at 20:35
9
$\begingroup$

Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage.

Data Leakage:

Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed.

You can read about it here : https://machinelearningmastery.com/data-leakage-machine-learning/

$\endgroup$
0
$\begingroup$

I think it doesn't matter whether you do it before or after since data leakage is only possible when classification results or your output information are also somehow flowing in the input model.

But since you are applying normalization on the input parameter and not the output no leakage could possibly happen.

$\endgroup$
0
$\begingroup$

Sklearn's own documentation indicates that:

"Note We here choose to illustrate data leakage with a feature selection step. This risk of leakage is however relevant with almost all transformations in scikit-learn, including (but not limited to) StandardScaler, SimpleImputer, and PCA."

"As with any other type of preprocessing, feature selection should only use the training data. Including the test data in feature selection will optimistically bias your model."

"10.2.2. How to avoid data leakage Below are some tips on avoiding data leakage:

Always split the data into train and test subsets first, particularly before any preprocessing steps."

https://scikit-learn.org/stable/common_pitfalls.html

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.