1
$\begingroup$

I am fairly new to machine learning. I came across the concept of Data Leakage. The article says that always split the data before performing preprocessing steps.

My question is, do steps such as discretization, grouping categories to a single category to reduce cardinality, converting categorical variables to binary variables, etc. lead to Data Leakage?

Should I split the data to train and test set before applying these steps?

Also, which are the main preprocessing steps I really need to be cautious of in order to avoid data leakage?

$\endgroup$

1 Answer 1

1
$\begingroup$

Think of the data being organized in rows (instances) and columns (features). Any pre-processing step which mixes information between/across rows can lead to data leakage.

A typical example is standardization or normalization. Applying min-max-scaling, for example, on the whole dataset would leak information since it is an aggregation across rows/instances. Another typical example is encoding categorical variables. When you train your encoder before the split you're taking into considerations values which you should not have seen before. Because when applied on real data, you might be presented unknown data values too.

The guiding principle here is, that your test strategy should resemble the real application in order to estimate the ability of your model to generalize to unseen data.

The safest way to avoid data leakage is to split the data before applying any pre-processing. Moreover, this approach supports designing the pipeline in a way that you can apply it to validation/test and later production data the same way you applied it on training data.

$\endgroup$
2
  • $\begingroup$ Thank you for your answer! $\endgroup$
    – Joe
    Commented May 16, 2021 at 12:26
  • $\begingroup$ @Joe glad it was helpful. If it answers your question please accept the answer. Otherwise your question will keep floating around as still open. $\endgroup$
    – Jonathan
    Commented May 16, 2021 at 13:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.