22
$\begingroup$

Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

normalized_X_features = pd.DataFrame(StandardScaler().fit_transform(X_features), columns = X_features.columns)
x_train, x_test, y_train, y_test = train_test_split(normalized_X_features, Y_feature, test_size=0.20, random_state=4)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(x_train, y_train)
y_test_pred = LR.predict(x_test)

Normalization after split

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)
normalized_x_train = pd.DataFrame(StandardScaler().fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(StandardScaler().fit_transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)

So far I have seen both approaches.

$\endgroup$
1
  • 2
    $\begingroup$ The 2nd approach is definitely wrong. You should not be calling a sklearn "fit" method on test data. 'fit' and 'fit_transform' are for training, 'predict' and 'transform' for testing. $\endgroup$ – David Waterworth Jul 3 '19 at 3:49
32
$\begingroup$

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.

[Precision thanks to Neil's comment] When normalizing the test set, one should apply the normalization parameters previously obtained from the training set as-is. Do not recalculate them on the test set, because they would be inconsistent with the model and this would produce wrong predictions.

$\endgroup$
5
  • 2
    $\begingroup$ And the test set must also be normalized (before prediction)? $\endgroup$ – Tauno Jul 2 '19 at 13:22
  • 3
    $\begingroup$ @TaunoTanilas you must treat the test set the same way you treat the training set. $\endgroup$ – Michael Hoffman Jul 2 '19 at 13:30
  • 3
    $\begingroup$ Yes, test set needs to be normalized before feeding the model. $\endgroup$ – Adhira Deogade Jul 2 '19 at 13:43
  • $\begingroup$ Important - and I suggest adding to this answer - whatever factors, offsets etc that you derive from the training set to normalise it should be applied as-is to normalise all data used from then on. Do not separately calculate mean, sd etc from the test set to normalise it starting from scratch. Re-use those values established from the training set. $\endgroup$ – Neil Slater Jun 30 at 20:31
  • $\begingroup$ @NeilSlater good point, done. $\endgroup$ – Erwan Jun 30 at 22:43
9
$\begingroup$

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like:

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)

scaler = StandardScaler()

normalized_x_train = pd.DataFrame(scaler.fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(scaler.transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)
$\endgroup$
1
  • $\begingroup$ Worth calling out that scaler.fit_transform is used on the training set, then scaler.transform is used on the test set as OP gets this wrong in the question. In addition, if this model will be re-used separately to the train, test run then the scaler's fitted params should be stored for re-use (I suppose you could store the training set and re-use it recalculate, but that's quite heavyweight for production use) $\endgroup$ – Neil Slater Jun 30 at 20:35
7
$\begingroup$

Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage.

Data Leakage:

Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed.

You can read about it here : https://machinelearningmastery.com/data-leakage-machine-learning/

$\endgroup$
0
$\begingroup$

I think it doesn't matter whether you do it before or after since data leakage is only possible when classification results or your output information are also somehow flowing in the input model.

But since you are applying normalization on the input parameter and not the output no leakage could possibly happen.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.