2
$\begingroup$

I understand why it is usefull to normalize data in general (at least I think I do). You take the mean and the standard deviation of the train data and apply it to both, the train and test data.

Why is it that we can not take the mean and std of the whole dataset combined (train and test data) and then normalize around these values?

$\endgroup$
1
$\begingroup$

The reason you split your dataset to Training and Test is to simulate real-world cases. What you actually do with the train-split validation is to evaluate your model in unknown data.

Imagine now that you have trained your model and you are on a production where new data keep coming for prediction. You might not get them in mass, but one by one such as in an API call. You don't have the mean and standard deviation of those "new" data. You only have the mean and std during the training process.

To sum up, train-test validation tries to be as close as possible to the real problem. And since you won't know anything about your upcoming data, you should not use any knowledge you get from the test data.

$\endgroup$
  • $\begingroup$ That makes perfect sense to me. Thank you very much. $\endgroup$ – nilleeee Feb 15 '19 at 9:48
  • $\begingroup$ Just as an additional note - if you were to normalize the test data on a different mean and std you would have a different normalization on both your train and test data which would not be good for your model. That is why you normalize on the train and then transform on the test. $\endgroup$ – Ethan Feb 16 '19 at 0:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.