Introduction

I understand the problem of data leakage that could be caused by the preprocessing step when our training and test sets are just samples of an unknown population. The preprocessing parameters should be calculated from the training set only, then we just apply the same procedure to validation/test set, since this would be the way to proceed with any other sample from the unknown population (in production stage, for example).

Question

What about the situation where we have the whole population at hand? Could we calculate the preprocessing parameters (scaling factors, encoding, etc.) from the entire population?

Extra Context

We have the whole population and the modeling process would depend of user input. The training set is defined by the user input and the trained model is used to classify the rest of the population.

Any time you use some input from the test set to make your model, you have a data leakage. Examples:

  • You calculate the average income according to some category of your users and add it as a feature, with the income for each user being an additional feature. This is a data leakage, as you are calculating these averages using data points on which you'll later make predictions. (In this case, you should calculate these averages using only the training data and/or other additional data that excludes our test set)
  • You calculate the average income for each user according to some category. Income is not part of your input variables and you took these averages from, say, census information. This is not a data leakage as you are not making predictions on the same data that you used to construct these averages.
  • Thank you for the answer, but I think it doesn't apply to my situation. I don't use the user input as data, it only defines the way I sample the population. – boechat107 Oct 9 at 18:04

Your Answer

 

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.