1
$\begingroup$

Introduction

I understand the problem of data leakage that could be caused by the preprocessing step when our training and test sets are just samples of an unknown population. The preprocessing parameters should be calculated from the training set only, then we just apply the same procedure to validation/test set, since this would be the way to proceed with any other sample from the unknown population (in production stage, for example).

Question

What about the situation where we have the whole population at hand? Could we calculate the preprocessing parameters (scaling factors, encoding, etc.) from the entire population?

Extra Context

We have the whole population and the modeling process would depend of user input. The training set is defined by the user input and the trained model is used to classify the population.

$\endgroup$
0
$\begingroup$

If you have the entire population, there is no need for inference. Thus data leakage is not an issue. You can fit any transformation on the data without a concern for its effect on prediction because there is no prediction step.

$\endgroup$
  • $\begingroup$ Yeah, this was also my conclusion. Thanks for the answer! $\endgroup$ – boechat107 Dec 10 '18 at 17:13
0
$\begingroup$

Any time you use some input from the test set to make your model, you have a data leakage. Examples:

  • You calculate the average income according to some category of your users and add it as a feature, with the income for each user being an additional feature. This is a data leakage, as you are calculating these averages using data points on which you'll later make predictions. (In this case, you should calculate these averages using only the training data and/or other additional data that excludes our test set)
  • You calculate the average income for each user according to some category. Income is not part of your input variables and you took these averages from, say, census information. This is not a data leakage as you are not making predictions on the same data that you used to construct these averages.
$\endgroup$
  • $\begingroup$ Thank you for the answer, but I think it doesn't apply to my situation. I don't use the user input as data, it only defines the way I sample the population. $\endgroup$ – boechat107 Oct 9 '18 at 18:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.