Data Snooping, Information Leakage When Performing Feature Normalization

Question

Assume that we have a training data set (with both features and labels) and a test data set (with only features).

When we build a machine learning model that requires normalization of the features, the correct way of doing normalization (to prevent information leakage) is to only use the training data set. Namely, a wrong way of doing normalization is to stack the training (exclude the Y column) and test data set together, and perform normalization (i.e., using the mean and variance of the entire training + test data set). The intuition here is very clear: if you want to get an unbiased estimator of your model performance in production, then when you train your model, you shouldn't instill any information in the test data set that will be used to gauge the actual performance of your model.

My question is the following: When we have trained the model correctly and about to use this model to do prediction, how should we normalize the test data set? I believe the correct way to normalize the test data set is: using the mean and variance obtained from the training data set to do normalization on the test data set.

However, why not just normalize the test data using its own mean and variance? Or why not stack train and test data set together and using the overall mean and variance to normalize the test data set? In the prediction stage, the idea and intuition of data snooping and information leakage is not clear to me.

Answer 1

The normalization parameters you fitted in training are now part of your model. You fitted the model weights on the training data: the normalization step is part of your model now, and the "parameters" for that step are the mean and variance of the training data, not the test data.

To help make this more concrete, let's pretend that this model is in production. You fit your model on whatever training data you had available, and now it's running as a scoring service. You can evaluate your model accuracy over time as data you've scored gets labeled to ground truth, but you're just evaluating the model not refitting it. How do you normalize these incoming predictions? You're going to need to use the mean and variance from whatever dataset you used to fit the model. If you'd rather do something like using the last K observations to estimate mean/variance, then that's the procedure you need to be testing when you fit the model.