6
$\begingroup$

Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

normalized_X_features = pd.DataFrame(StandardScaler().fit_transform(X_features), columns = X_features.columns)
x_train, x_test, y_train, y_test = train_test_split(normalized_X_features, Y_feature, test_size=0.20, random_state=4)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(x_train, y_train)
y_test_pred = LR.predict(x_test)

Normalization after split

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)
normalized_x_train = pd.DataFrame(StandardScaler().fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(StandardScaler().fit_transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)

So far I have seen both approaches.

$\endgroup$
  • $\begingroup$ The 2nd approach is definitely wrong. You should not be calling a sklearn "fit" method on test data. 'fit' and 'fit_transform' are for training, 'predict' and 'transform' for testing. $\endgroup$ – David Waterworth Jul 3 at 3:49
11
$\begingroup$

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.

$\endgroup$
  • $\begingroup$ And the test set must also be normalized (before prediction)? $\endgroup$ – Tauno Jul 2 at 13:22
  • 2
    $\begingroup$ @TaunoTanilas you must treat the test set the same way you treat the training set. $\endgroup$ – Michael Hoffman Jul 2 at 13:30
  • $\begingroup$ Yes, test set needs to be normalized before feeding the model. $\endgroup$ – Adhira Deogade Jul 2 at 13:43
3
$\begingroup$

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like:

x_train, x_test, y_train, y_test = train_test_split(X_features, Y_feature, test_size=0.20, random_state=4)

scaler = StandardScaler()

normalized_x_train = pd.DataFrame(scaler.fit_transform(x_train), columns = x_train.columns)
LR = LogisticRegression(C=0.01, solver='liblinear').fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(scaler.transform(x_test), columns = x_test.columns)
y_test_pred = LR.predict(normalized_x_test)
$\endgroup$
1
$\begingroup$

Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage.

Data Leakage:

Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed.

You can read about it here : https://machinelearningmastery.com/data-leakage-machine-learning/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.