Resampling for imbalaced datasets: should testing set also be resampled?

Question

Apologies for what is probably a basic question but I have not been able to find a definitive answer either in the literature or in the Internet.

When dealing with an imbalanced dataset one possible strategy is to resample either the minority or the majority class to artificially generate a balanced training set that can be used to train a machine learning model.

My doubt stems from the fact that a testing set is supposed to be a representation of what real-world data is going to look like. Under this assumption, my understanding is that the testing set is not to be resampled, unlike the training set, since the imbalance we are trying to deal with is present in the live data in the first place.

Could someone please clarify if this intuition is right?

Answer 1

The resampling of the training data is to better represent the minority class so your classifier would have more samples to learn from (Oversampling) or less samples to better differientiate your minority class samples from the rest (Undersampling). Not only your test data must be untouched during oversampling or undersampling but also your validation data. One logical argument that prevents you from touching your test data is that in a real-world scenario, you wouldn't have access to the target variable ( that's what you want to predict ) and in order to perform resampling, you need to know which class a sample belongs to for you to remove it (undersampling) or find it's nearest neighbor(s) (oversampling)

Example of an oversampling during cross-validation just below : What i'm basically doing here to avoid leaking information from trainset to testset( and valset ), every iteration, at each fold, i oversample the remaining folds, train a model with the oversampled new trainset, get my preds, and iterate over and over again. Each time i get a new fold for validation, i oversample all the others, and get predictions for that validation fold.

for ind,(ind_train,ind_val) in (enumerate (kfolds.split(X,y))): # Stratified Kfold
   X_train,X_val = X.iloc[ind_train],X.iloc[ind_val] 
   y_train,y_val = y.iloc[ind_train],y.iloc[ind_val]
   sm = SMOTE(random_state=12, ratio = 1.0)
   X_train_res, y_train_res = sm.fit_sample(X_train, y_train)##oversampled trainset
   xgb = XGBClassifier(max_depth=5,colsample_bytree=0.9,min_child_weight=2,learning_rate=0.09,objective = "binary:logistic",n_estimators=148)
   xgb.fit(X_train_res,y_train_res)
   val_pred = xgb.predict(X_val) ##out of fold predictions on my validation set
   train_pred = xgb.predict(X_train)##oof preds on my trainset
   test_pred = xgb.predict(X_test)##oof preds on my whole test set

Answer 2

You are right! As you said, the final goal of the model is to work with real life data distribution, therefor the test data has to stay unchanged.

The goal of resampling is not to have the same number of objects from all classes, but to fit model to work with your data. So an intuitive example might be: if you have k objects of the first class and 5*k of the second, but your model learn data well, then resampling is not necessary. It's the reason, why we shouldn't use it during test (we have already trained model)

One more thing that could be useful to remember is that F1 score could be calculate in multiclass task in different ways (it could account label imbalance), here is an example from sklearn https://scikit-learn.org/stable/modules/generated/sklearn.metrics.f1_score.html