I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data.

Why is it that we can not take the mean and std of the whole dataset combined (train and test data) and then normalize around these values?


1 Answer 1


The reason you split your dataset to Training and Test is to simulate real-world cases. What you actually do with the train-split validation is to evaluate your model in unknown data.

Imagine now that you have trained your model and you are on a production where new data keep coming for prediction. You might not get them in mass, but one by one such as in an API call. You don't have the mean and standard deviation of those "new" data. You only have the mean and std during the training process.

To sum up, train-test validation tries to be as close as possible to the real problem. And since you won't know anything about your upcoming data, you should not use any knowledge you get from the test data.

  • $\begingroup$ That makes perfect sense to me. Thank you very much. $\endgroup$
    – nilleeee
    Commented Feb 15, 2019 at 9:48
  • 1
    $\begingroup$ Just as an additional note - if you were to normalize the test data on a different mean and std you would have a different normalization on both your train and test data which would not be good for your model. That is why you normalize on the train and then transform on the test. $\endgroup$
    – Ethan
    Commented Feb 16, 2019 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.