1
$\begingroup$

I found in multiple sources the recommendation not to fit the normalization parameters on the combined train/test dataset when evaluating the model, to prevent data leakage.

I presume this recommendation assumes prediction data would flow to the model online, and the entire data is not known in advance.

Does this recommendation also apply to when the entire data is known in advance? And even if not, I guess it is possible to refit the model every time we get new data, with normalization parameters acheived from the entire dataset seen until now?

What are the pros and cons of normalizing over the entire dataset before fitting (as opposed to only on the train set)?

Improve this question
$\endgroup$
0
$\begingroup$

The point of evaluating your model is to assess how it will perform when it is used on unseen data. Normalizing over the entire dataset causes a data leakage (information coming from the test set). As a result, you will get a biased evaluation of your model.

In any machine learning task, the end goal is to use your model on unseen data to see how much the model has learned and how well it can generalize to unseen data. In that regard, there is no pros for normalizing over the entire dataset, and you should avoid doing so!

Improve this answer
$\endgroup$
  • $\begingroup$ I'm talking about the case where there is no unseen data. The entire dataset is known in advance, but only a part of it is tagged. In this case, normalizing over the entire dataset is really causing leakage? I cannot think of a reason why it would... $\endgroup$ – Itay Alfia Jun 22 '19 at 22:05
  • $\begingroup$ To make this more concrete: lets say I have a database of 10,000 images. Some are annotated and the rest are not. I want to generalize annotations on all the images from the few annotated ones, and I would never need to apply the model on other images from the ones I already have. And even if the need would arise, I can retrain the model again with new recomputed normalization parameters - if I would get any benefit from this, which I'm not sure... $\endgroup$ – Itay Alfia Jun 22 '19 at 22:14
  • $\begingroup$ My task is different from the standard machine learning task - I want to generalize to untagged seen data, which I know in advance at train time and I can make use of if necessary... $\endgroup$ – Itay Alfia Jun 22 '19 at 22:17
  • $\begingroup$ If the data in the test set is not labeled, then it is not an actual test set (you can't test the performance of your algorithm), and besides, unseen data is basically the same as unlabeled data. I understand what you mean, just know that the whole issue of data leakage is related to the assessment of your model performance on unlabeled data: if you normalize on the whole dataset, you are likely to get a biased estimate of the model performance. $\endgroup$ – astiegler Jun 24 '19 at 14:07
  • $\begingroup$ Usually in practice, the difference is likely not to be significant, but I would not recommend doing that anyway because you probably won't gain anything in terms of performance by doing it $\endgroup$ – astiegler Jun 24 '19 at 14:09

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.