3
$\begingroup$

I have recently started using kaggle and I have stumbled on a few examples of practices I would consider do be data leakage. Many of them were done by people well established on the platform and I could tell by their notebooks, that they knew what they were doing.

As some examples I have seen someone fix skewness on the whole dataset before any train-test split. As another I have seen multiple people impute missing data not only based on the whole dataset, but also taking into consideration the labels of the observations. So they imputed one value for one class and another for the second class. Why? Is this not data leakage? Shouldn't practices like this be avoided? Am I missing something here?

I have found this question, but I don't believe it is applicable in the cases I wrote about, since the final goal of all the notebooks was to create predictive models.

Edit:

I realized that a log transformation would be fine to apply to the whole dataset, when fixing skewness, but that is not the case in the example I was talking about. The transformation applied was box-cox.

The specific transformations in the examples about imputation were all filling the missing values with the means of features with respect to the class labels.

Improve this question
$\endgroup$
4
  • $\begingroup$ Can you elaborate on what type of skewness adjustments and imputing you're exactly referring to? $\endgroup$
    – Jonathan
    Commented Jul 18, 2021 at 12:05
  • $\begingroup$ Sure. Both transformations were applied to the whole dataset before any train-test split. In the one with skewness someone uses a box-cox transformation. In the many I have seen with imputation people use mean value of some feature in a particular class to impute the missing values. So they impute in the same column value x_1 for class A and value x_2 for class B, both of which are means with respect to the class labels. $\endgroup$
    – Mateusz
    Commented Jul 18, 2021 at 12:28
  • 2
    $\begingroup$ Did they use the whole data set to choose the Box-Cox parameter, or did they just apply a transformation to the entire data set at once? As long as you don’t look at the out-of-sample data to pick a hyperparameter, it is fine to hit everything with a transformation in one programming step. For instance, it might be known from previous work that a particular Box-Cox transform is helpful. // All that said, I share your concerns about data leakage, particularly when it comes to how that by-class imputation occurs. $\endgroup$
    – Dave
    Commented Jul 18, 2021 at 12:41
  • $\begingroup$ Yeah, the thing is they did use the whole data set to choose the box-cox parameter. I was expecting there would be some explanation for those practices I was not aware of, especially since many of those works were done by people I could say were quite knowledgeable, but well. Thank you for the answer. $\endgroup$
    – Mateusz
    Commented Jul 18, 2021 at 13:25

1 Answer 1

Reset to default
3
$\begingroup$

If you have the whole population and do not want to predict anything, as stated in the question link you gave, then it is fine to use the whole population for preprocessing as it will give better results during training (which is our only goal).

But if you do not have the whole population and want to do predictions, it is best to keep the test set untouched until the very end when you are sure the model you have built on the train and valid sets is the best you could have built with the info you had.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.