Why you shouldn't upsample before cross validation

Question

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross-validate when oversampling data using SMOTE technique.

I have created a model using AdaBoost algorithm and set the following parametres to be used in Grid Search:

    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

According to the article, this is the wrong way to oversample:

    X_train_upsample, y_train_upsample = SMOTE(random_state=42).fit_sample(X_train, y_train)
    
    # cross-validate using grid search
    
    grid_naive_up = GridSearchCV(ada, param_grid=params, cv=kf, 
                                 scoring='recall').fit(X_train_upsample, 
                                                       y_train_upsample)
    grid_naive_up.best_score_

0.6715940782827282

    # test set
    recall_score(y_test, grid_naive_up.predict(X_test))

0.2824858757062147

Whereas the correct way to oversample is like so:

    from imblearn.pipeline import Pipeline, make_pipeline
    
    imba_pipeline = make_pipeline(SMOTE(random_state=42), 
                                  AdaBoostClassifier(n_estimators=100, random_state=42))
    cross_val_score(imba_pipeline, X_train, y_train, scoring='recall', cv=kf)
    new_params = {'adaboostclassifier__' + key: params[key] for key in params}
    grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring='recall',
                            return_train_score=True)
    grid_imba.fit(X_train, y_train);


    # How well do we do on our validation set?
    grid_imba.best_score_

0.29015614186873506

    # compare this to the test set:
    y_test_predict = grid_imba.predict(X_test)

0.2824858757062147

So, according to the article, the first method is wrong because when upsampling before cross validation, the validation recall isn't a good measure of the test recall (28.2%). However, when using the imblearn pipeline for upsampling as part of the cross validation, the validation set recall (29%) was a good estimate of the test set recall (28.3%). According to the article, the reason for this is:

When upsampling before cross validation, you will be picking the most oversampled model, because the oversampling is allowing data to leak from the validation folds into the training folds.

Can anyone explain to me simply how the oversampling allows data to leak into the validation and causes the overfitting? And why does this problem not occur in the imblearn pipeline?

Answer 1

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE.

The first reason why such a procedure is invalid is that, this way, some of the duplicates due to upsampling will end up both to the training and the validation splits (CV folds); the result being that the algorithm is validated with some samples that have already been seen during training, which invalidates the very fundamental requirement of a validation set (fold) and it is actually the very definition of data leakage. For more details, see own answer in the SO thread Process for oversampling data for imbalanced binary classification; quoting from there:

I once witnessed a case where the modeller was struggling to understand why he was getting a ~ 100% test accuracy, much higher than his training one; turned out his initial dataset was full of duplicates -no class imbalance here, but the idea is similar- and several of these duplicates naturally ended up in his test set after the split, without of course being new or unseen data...

But there is also a second reason: this procedure shows biased performance measures in our validation folds that are no longer representative of reality: remember, we want our validation folds to be representative of the real unseen data, which of course will be imbalanced. Performing CV after upsampling results also to artificially balancing our validation folds; doing so, and claiming that we get X% accuracy when a great part of this accuracy will be due to the artificially upsampled minority class makes no sense, and gives misleading impressions. For details, see own answer in the SO thread Balance classes in cross validation. Notice that the author of the post you have linked to says (rather cryptically, and only in a parenthesis):

(we are smart enough not to oversample the test data)

For more corroboration, here is Max Kuhn, creator of the caret R package and co-author of the (highly recommended) Applied Predictive Modelling textbook, in Chapter 11: Subsampling For Class Imbalances of the caret ebook:

You would never want to artificially balance the test set; its class frequencies should be in-line with what one would see “in the wild”.

---

Now, it's true that the above hold for the case of balancing through simple upsampling of the minority class; but SMOTE does not do that - it uses interpolation to create synthetic samples that are "close" enough to the real minority ones. How does this change the situation?

Not much.

• The second reason stated above (biased performance measures in the validation folds) is still fully applicable - in fact, it holds independently of the exact nature of the upsampling (duplicate samples or synthetic ones).

• Given that the synthetic samples generated by SMOTE are indeed highly correlated with the real ones, the problems due to the first reason mentioned above are still largely present, although somewhat ameliorated.

In contrast, the pipeline approach does not suffer from these issues, because it first splits into training and validation folds, and applies SMOTE subsequently only to the training ones.