1
$\begingroup$

I came across the paper "Leakage and the Reproducibility Crisis in ML-based Science" by Sayash Kapoor and Arvind Narayanan, wherein the authors argue that both over- and under-sampling the entire dataset can lead to data leakage. When it comes to imputing or oversampling, the concept is relatively straightforward; consider that we decide to use linear regression for these purposes. The oversampled data, drawn from both the training and test sets, will have a relationship with the chosen linear regression model. As a result, we are creating an artificial relationship that was not present in the original data, likely leading to inflated performance indicators.

On the other hand, the reasons why undersampling might lead to leakage are not as obvious to me. After all, doesn't undersampling simply mean that we are working with fewer observations? I am puzzled as to why working with a reduced dataset results in leakage, which traditionally occurs when the model is inadvertently trained with data from the test set.

$\endgroup$

2 Answers 2

1
$\begingroup$

The paper you reference is using a different definition of leakage than the one you stated above: "the model is inadvertently trained with data from the test set". At the start of section 2.1 they define data leakage as:

Data leakage is a spurious relationship between the independent variables and the target variable that arises as an artifact of the data collection, sampling, or pre-processing strategy. Since the spurious relationship won’t be present in the distribution about which scientific claims are made, leakage usually leads to inflated estimates of model performance.

In other words, data leakage is anything done to the test data that means it is no longer representative of the distribution it is drawn from.

So undersampling the full dataset before splitting it into test/training sets is a problem because it changes the class distribution in the test data. The results will therefore be biased due to the skewed class proportions in the test data.

$\endgroup$
1
$\begingroup$

When talking about train and test sets in this case you are referring to the folds, k-1 folds are used for training the model, and the remaining fold is used for testing. You can now treat these as a regular train/test split, meaning you should definitely not be over/under-sampling using the test set, because that's cheating.

You can however do whatever you want on the train set. So to do cross-validation right you need to do over/under-sampling just based on the k-1 folds while leaving the remaining fold untouched. So this, if done, needs to be done inside the cross-validation loop and not outside of it - before.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.