# Imbalanced Dataset: Train/test split before and after SMOTE

This question is similar but different from my previous one. I have a binary classification task related to customer churn for a bank. The dataset contains 10,000 instances and 11 features. The target variable is imbalanced (80% remained as customers (0), 20% churned (1)).

Initially, I followed this approach: I first split the dataset into training and test sets, while preserving the 80-20 ratio for the target variable in both sets. I keep 8,000 instances in the training set and 2,000 in the test set. After pre-processing, I address the class imbalance in the training set with SMOTEENN:

from imblearn.combine import SMOTEENN

smt = SMOTEENN(random_state=random_state)
X_train, y_train = smt.fit_sample(X_train, y_train)


Now, my training set has 4774 1s and 4182 0s. I know proceed to building ML models. I use scikit-learn’s GridSearchCV with cv = KFold(n_splits=5, shuffle=True, random_state=random_state) and optimise based on the recall score. For instance, for a Random Forest Classifier:

cv = KFold(n_splits=5, shuffle=True, random_state=random_state)
scoring_metric='recall'

rf = RandomForestClassifier(random_state=random_state)
param_grid = {
'n_estimators': [100],
'criterion': ['entropy', 'gini'],
'bootstrap': [True, False],
'max_depth': [6],
'max_features': ['auto', 'sqrt'],
'min_samples_leaf': [2, 3, 5],
'min_samples_split': [2, 3, 5]
}

rf_clf = GridSearchCV(estimator=rf,
param_grid=param_grid,
scoring=scoring_metric,
cv=cv,
verbose=False,
n_jobs=-1)

best_rf_clf = rf_clf.fit(X_train, y_train)

y_pred = cross_val_predict(best_rf_clf.best_estimator_,X_train, y_train,cv=cv)
print('Train: ', np.round(recall_score(y_train, y_pred), 3))

y_pred = best_rf_clf.best_estimator_.fit(X_train, y_train).predict(X_test)
print(' Test: ', np.round(recall_score(y_test, y_pred), 3))


My recall CV score on the training set is 0.902, while the score on the test is 0.794.

However, when apply SMOTEENN on the full dataset and then split into training and test sets, I get a recall CV score on the training set equal to 0.913, and 0.898 for the test set.

How can we explain this difference between the two approaches? What causes this gap between the two sets in the first approach (split then SMOTEENN) compared to the second one (SMOTEENN and then split)? My guess is that the second approach leads to a more balanced test set (1220 1s, 1036 0s), compared to the first one (1607 1s, 393 0s). Thanks!

Essentially applying SMOTE makes the job easier for the model: SMOTE generates artificial instances which tend to have the same properties as each other, so it's easier for the model to capture their patterns. However these instances are rarely a good representative sample for the minority class, so there's a higher risk that the model overfits.

Of course if SMOTE is also applied to the test set, the model appears to perform better. This is the equivalent of changing a difficult question to an easier one in order to answer the question better.

Resampling methods are rarely a good solution to the imbalance problem. It's important to understand that imbalanced data is a problem only because the minority class in the training set is not representative enough and/or the features are not good enough indicators for the label. The ideal scenario is to solve these two problems, then the model can perform perfectly well despite the imbalance.

• Thanks for giving another good explanation, Erwan! I will do more reading on dealing with imbalanced datasets after your answer. What would you suggest apart from SMOTE? I have performed Undersampling to bring the ratio to 50:50, but I feel I am discarding a lot of useful information. Nov 24, 2021 at 11:59
• @KL_o7 It would help if you could explain why class imbalance is an issue for you. I claim that class imbalance almost certainly is not a problem, and there is no need to solve a non-problem. stats.stackexchange.com/questions/357466 fharrell.com/post/class-damage fharrell.com/post/classification stats.stackexchange.com/a/359936/247274 stats.stackexchange.com/questions/464636 twitter.com/f2harrell/status/1062424969366462473?lang=en
– Dave
Nov 24, 2021 at 13:45
• @KK_o7 "I have performed Undersampling to bring the ratio to 50:50, but I feel I am discarding a lot of useful information." You can build an ensemble of models, they all have the same minority class, but different under-sampled majority class, this way you don't throw away the information. Nov 24, 2021 at 21:13
• If you want a good recall score, why not just label everything the case that you’re trying to detect? (I realize why this is (probably) ridiculous, but how you respond will be informative.) // When you read the material I posted, you’ll see why hard classifications that result in metrics like recall actually are not as desirable as it might appear, and why many (like Frank Harrell) advocate for direct assessments of the probability outputs of models.
– Dave
Nov 24, 2021 at 23:38
• @KK_o7 I'd suggest trying to understand why errors happen instead of focusing on treating the imbalance. There are many ways to do that and it depends a lot on the data. For example you could pick a few samples randomly and investigate what happens in each case, or use clustering to observe the different profiles of customer. Errors might also come from poor quality in the data, in this case some preprocessing might help. Nov 25, 2021 at 15:11

You must apply SMOTE after splitting into training and test, not before. Doing SMOTE before is bogus and defeats the purpose of having a separate test set.

At a really crude level, SMOTE essentially duplicates some samples (this is a simplification, but it will give you a reasonable intuition). If you duplicate every sample ten times, and then split into train and test, then about 5 copies of each sample will appear in the training set and about 5 copies in the test set. Effectively, the test set becomes more or less identical to the training set. A classifier that just memorizes the training set (and overfits to it massively) will also do very well on the test set -- but it would perform terribly in practice.

So duplicating samples before splitting into training/test is effectively evaluating your classifier on the training set, which we know is a biased measure of its performance. Doing SMOTE before splitting will have similar bad properties.

• Thanks a lot for the nice explanation, D.W.! Nov 30, 2021 at 14:11
• @D.W I would have to disagree on the fact that smote duplicates samples. SMOTE identifies the k nearest neighbors of the data points from the minority class and it creates a new point at a random location between all the neighbors. These new points represent artificial data that belong to the minority class. May 4 at 14:27
• @MalekKamoua, I'm familiar with SMOTE. I didn't say it duplicates samples. May I direct your attention to where I wrote that "At a really crude level, SMOTE essentially duplicates" and "this is a simplification, but it will give you a reasonable intuition"? It's easy to understand why duplicating samples before splitting is flawed. SMOTE does something more complicated that's harder to reason about, but has a similar problem: doing SMOTE before splitting can have a similar problem as duplicating before splitting (not as severe or easy to understand but the potential is still there).
– D.W.
May 4 at 17:11