5
$\begingroup$

I have an imbalanced dataset and I am trying different methods to address the data imbalance. I found this article that explains the correct way to cross validate when oversampling data using SMOTE technique.

I have created a model using AdaBoost algorithm and set the following parametres to be used in Grid Search:

    ada = AdaBoostClassifier(n_estimators=100, random_state=42)
    params = {
        'n_estimators': [50, 100, 200],
        'random_state': [42]
    }

According to the article, this is the wrong way to oversample:

    X_train_upsample, y_train_upsample = SMOTE(random_state=42).fit_sample(X_train, y_train)
    
    # cross-validate using grid search
    
    grid_naive_up = GridSearchCV(ada, param_grid=params, cv=kf, 
                                 scoring='recall').fit(X_train_upsample, 
                                                       y_train_upsample)
    grid_naive_up.best_score_

0.6715940782827282

    # test set
    recall_score(y_test, grid_naive_up.predict(X_test))

0.2824858757062147

Whereas the correct way to oversample is like so:

    from imblearn.pipeline import Pipeline, make_pipeline
    
    imba_pipeline = make_pipeline(SMOTE(random_state=42), 
                                  AdaBoostClassifier(n_estimators=100, random_state=42))
    cross_val_score(imba_pipeline, X_train, y_train, scoring='recall', cv=kf)
    new_params = {'adaboostclassifier__' + key: params[key] for key in params}
    grid_imba = GridSearchCV(imba_pipeline, param_grid=new_params, cv=kf, scoring='recall',
                            return_train_score=True)
    grid_imba.fit(X_train, y_train);


    # How well do we do on our validation set?
    grid_imba.best_score_

0.29015614186873506

    # compare this to the test set:
    y_test_predict = grid_imba.predict(X_test)

0.2824858757062147

So, according to the article, the first method is wrong because when upsampling before cross validation, the validation recall isn't a good measure of the test recall (28.2%). However when using the imblearn pipeline for upsampling as part of the cross validation, the validation set recall (29%) was a good estimate of the test set recall (28.3%). According to the article, the reason for this is:

When upsampling before cross validation, you will be picking the most oversampled model, because the oversampling is allowing data to leak from the validation folds into the training folds.

Can anyone explain to me simply how the oversampling allows data to leak into the validation and causes the overfitting? And why does this problem not occur in the imblearn pipeline?

$\endgroup$
6
  • 1
    $\begingroup$ My guess is that in the first case, your training dataset has been artificially balanced, so you're estimation of performance of your model is on a dataset that does not follow the original distribution of your data - you also estimate your score on data that is synthetic which seems improper - the grid search val split doesn't know that some data is sort of duplicated by SMOTE therefore the original sample might be in train and the oversampled version might get into the val set. In the second case though, I imagine that the pipeline takes care for you of only oversampling the training set. $\endgroup$ – mprouveur Sep 22 '20 at 15:09
  • 1
    $\begingroup$ or at least oversampling each set only after the train/val split has been done $\endgroup$ – mprouveur Sep 22 '20 at 15:10
  • 5
    $\begingroup$ Does this answer your question? Why did sampling boost the performance of my model?, also Oversampling before Cross-Validation, is it a problem? $\endgroup$ – Ben Reiniger Sep 22 '20 at 16:04
  • $\begingroup$ @BenReiniger to be very precise, and despite the similarity in the titles, none of the linked questions is about SMOTE, which is actually the case here $\endgroup$ – desertnaut Sep 23 '20 at 12:08
  • $\begingroup$ The voted duplicate is about SMOTE, albeit only after a few comments. $\endgroup$ – Ben Reiniger Sep 23 '20 at 13:54
7
$\begingroup$

To see clearly why the procedure of upsampling before CV is mistaken and it leads to data leakage and other undesired consequences, it is useful to imagine first the simpler "baseline" case, where we simply upsample (i.e. create duplicate samples) without SMOTE.

The first reason why such a procedure is invalid is that, this way, some of the duplicates due to upsampling will end up both to the training and the validation splits (CV folds); the result being that the algorithm is validated with some samples that have already been seen during training, which invalidates the very fundamental requirement of a validation set (fold) and it is actually the very definition of data leakage. For more details, see own answer in the SO thread Process for oversampling data for imbalanced binary classification; quoting from there:

I once witnessed a case where the modeller was struggling to understand why he was getting a ~ 100% test accuracy, much higher than his training one; turned out his initial dataset was full of duplicates -no class imbalance here, but the idea is similar- and several of these duplicates naturally ended up in his test set after the split, without of course being new or unseen data...

But there is also a second reason: this procedure shows biased performance measures in our validation folds that are no longer representative of reality: remember, we want our validation folds to be representative of the real unseen data, which of course will be imbalanced. Performing CV after upsampling results also to artificially balancing our validation folds; doing so, and claiming that we get X% accuracy when a great part of this accuracy will be due to the artificially upsampled minority class makes no sense, and gives misleading impressions. For details, see own answer in the SO thread Balance classes in cross validation. Notice that the author of the post you have linked to says (rather cryptically, and only in a parenthesis):

(we are smart enough not to oversample the test data)

For more corroboration, here is Max Kuhn, creator of the caret R package and co-author of the (highly recommended) Applied Predictive Modelling textbook, in Chapter 11: Subsampling For Class Imbalances of the caret ebook:

You would never want to artificially balance the test set; its class frequencies should be in-line with what one would see “in the wild”.


Now, it's true that the above hold for the case of balancing through simple upsampling of the minority class; but SMOTE does not do that - it uses interpolation to create synthetic samples that are "close" enough to the real minority ones. How does this change the situation?

Not much.

  • The second reason stated above (biased performance measures in the validation folds) is still fully applicable - in fact, it holds independently of the exact nature of the upsampling (duplicate samples or synthetic ones).

  • Given that the synthetic samples generated by SMOTE are indeed highly correlated with the real ones, the problems due to the first reason mentioned above are still largely present, although somewhat ameliorated.

In contrast, the pipeline approach does not suffer from these issues, because it first splits into training and validation folds, and applies SMOTE subsequently only to the training ones.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.