Imbalanced classification task – Discrepancy between learning curves and test set evaluation

Question

I have a binary classification task related to customer churn for a bank. The dataset contains 10,000 instances and 11 features. The target variable is imbalanced (80% remained as customers (0), 20% churned (1)).

My approach is the following: I first split the dataset into training and test sets, while preserving the 80-20 ratio for the target variable in both sets. I keep 8,000 instances in the training set and 2,000 in the test set. After pre-processing, I address the class imbalance in the training set with SMOTEENN:

from imblearn.combine import SMOTEENN

smt = SMOTEENN(random_state=random_state)
X_train, y_train = smt.fit_sample(X_train, y_train)

Now, my training set has 4774 1s and 4182 0s. I know proceed to building ML models. I use scikit-learn’s GridSearchCV with cv = KFold(n_splits=5, shuffle=True, random_state=random_state) and optimise based on the recall score. For instance, for a Random Forest Classifier:

cv = KFold(n_splits=5, shuffle=True, random_state=random_state)
scoring_metric='recall'

rf = RandomForestClassifier(random_state=random_state)
param_grid = {
    'n_estimators': [100],
    'criterion': ['entropy', 'gini'],
    'bootstrap': [True, False],
    'max_depth': [6],
    'max_features': ['auto', 'sqrt'],
    'min_samples_leaf': [2, 3, 5],
    'min_samples_split': [2, 3, 5]
}

rf_clf = GridSearchCV(estimator=rf,
                      param_grid=param_grid,
                      scoring=scoring_metric,
                      cv=cv,
                      verbose=False,
                      n_jobs=-1)

best_rf_clf = rf_clf.fit(X_train, y_train)

I get the following confusion matric and learning curves (based on this link):

Notice that the cross-validation curve exceeds 90% when training includes all samples. However, when I got to the (pre-processed, imbalanced) test set, I get a significantly lower recall score around 79%.

y_pred = best_rf_clf.best_estimator_.fit(X_train, y_train).predict(X_test)

recall_score(y_test, y_pred)
# outputs 0.793

What can explain this difference? I have shuffled both sets to make sure nothing funny happens when splitting them. Apart from that, I cannot think of anything else. Thanks a lot!!

Answer 1

Training on the resampled data causes the model to assign roughly the same proportion of positive and negative labels, especially in difficult cases where it doesn't have clear enough indications from the features.

When testing on the test set with the original distribution (this is the correct method of course), the model applies what it learned: for "difficult cases", it assigns roughly half positive labels. But the test set has a higher proportion of positive cases, so this results in more false negative errors and therefore lower recall than on the training set.