Data normalization before or after train-test split?

Which one is the right approach to make data normalization - before or after train-test split?

Normalization before split

normalized_X_features = pd.DataFrame(
StandardScaler().fit_transform(X_features),
columns = X_features.columns
)
x_train, x_test, y_train, y_test = train_test_split(
normalized_X_features,
Y_feature,
test_size=0.20,
random_state=4
)
LR = LogisticRegression(
C=0.01,
solver='liblinear'
).fit(x_train, y_train)

y_test_pred = LR.predict(x_test)


Normalization after split

x_train, x_test, y_train, y_test = train_test_split(
X_features,
Y_feature,
test_size=0.20,
random_state=4
)
normalized_x_train = pd.DataFrame(
StandardScaler().fit_transform(x_train),
columns = x_train.columns
)
LR = LogisticRegression(
C=0.01,
solver='liblinear'
).fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(
StandardScaler().fit_transform(x_test),
columns = x_test.columns
)
y_test_pred = LR.predict(normalized_x_test)


So far I have seen both approaches.

• The 2nd approach is definitely wrong. You should not be calling a sklearn "fit" method on test data. 'fit' and 'fit_transform' are for training, 'predict' and 'transform' for testing. Jul 3, 2019 at 3:49
• For sure after split what techniques you are going to apply for training set,apply the same on test set as well. Jan 5 at 11:44

4 Answers

Normalization across instances should be done after splitting the data between training and test set, using only the data from the training set.

This is because the test set plays the role of fresh unseen data, so it's not supposed to be accessible at the training stage. Using any information coming from the test set before or during training is a potential bias in the evaluation of the performance.

[Precision thanks to Neil's comment] When normalizing the test set, one should apply the normalization parameters previously obtained from the training set as-is. Do not recalculate them on the test set, because they would be inconsistent with the model and this would produce wrong predictions.

• And the test set must also be normalized (before prediction)? Jul 2, 2019 at 13:22
• @TaunoTanilas you must treat the test set the same way you treat the training set. Jul 2, 2019 at 13:30
• Yes, test set needs to be normalized before feeding the model. Jul 2, 2019 at 13:43
• Important - and I suggest adding to this answer - whatever factors, offsets etc that you derive from the training set to normalise it should be applied as-is to normalise all data used from then on. Do not separately calculate mean, sd etc from the test set to normalise it starting from scratch. Re-use those values established from the training set. Jun 30, 2021 at 20:31
• @StackOverflow no, it's not always needed to normalize anything actually, it depends on the specifics of the task,data and algorithm. You should ask a new question if you want to know about a particular case. Apr 7 at 12:26

As @Erwan said, you should normalize the training set and then use the same normalization steps on the test set. So your code should look like:

x_train, x_test, y_train, y_test = train_test_split(
X_features,
Y_feature,
test_size=0.20,
random_state=4)

scaler = StandardScaler()

normalized_x_train = pd.DataFrame(
scaler.fit_transform(x_train),
columns = x_train.columns
)

LR = LogisticRegression(
C=0.01,
solver='liblinear'
).fit(normalized_x_train, y_train)

normalized_x_test = pd.DataFrame(
scaler.transform(x_test),
columns = x_test.columns
)
y_test_pred = LR.predict(normalized_x_test)

• Worth calling out that scaler.fit_transform is used on the training set, then scaler.transform is used on the test set as OP gets this wrong in the question. In addition, if this model will be re-used separately to the train, test run then the scaler's fitted params should be stored for re-use (I suppose you could store the training set and re-use it recalculate, but that's quite heavyweight for production use) Jun 30, 2021 at 20:35

Answer to your question: Do Normalization after splitting into train and test/validation. The reason is to avoid any data leakage.

Data Leakage:

Data leakage is when information from outside the training dataset is used to create the model. This additional information can allow the model to learn or know something that it otherwise would not know and in turn invalidate the estimated performance of the mode being constructed.

You can read about it here : https://machinelearningmastery.com/data-leakage-machine-learning/

I think it doesn't matter whether you do it before or after since data leakage is only possible when classification results or your output information are also somehow flowing in the input model.

But since you are applying normalization on the input parameter and not the output no leakage could possibly happen.