# Imbalanced classification task – Discrepancy between learning curves and test set evaluation

I have a binary classification task related to customer churn for a bank. The dataset contains 10,000 instances and 11 features. The target variable is imbalanced (80% remained as customers (0), 20% churned (1)).

My approach is the following: I first split the dataset into training and test sets, while preserving the 80-20 ratio for the target variable in both sets. I keep 8,000 instances in the training set and 2,000 in the test set. After pre-processing, I address the class imbalance in the training set with SMOTEENN:

from imblearn.combine import SMOTEENN

smt = SMOTEENN(random_state=random_state)
X_train, y_train = smt.fit_sample(X_train, y_train)


Now, my training set has 4774 1s and 4182 0s. I know proceed to building ML models. I use scikit-learn’s GridSearchCV with cv = KFold(n_splits=5, shuffle=True, random_state=random_state) and optimise based on the recall score. For instance, for a Random Forest Classifier:

cv = KFold(n_splits=5, shuffle=True, random_state=random_state)
scoring_metric='recall'

rf = RandomForestClassifier(random_state=random_state)
param_grid = {
'n_estimators': [100],
'criterion': ['entropy', 'gini'],
'bootstrap': [True, False],
'max_depth': [6],
'max_features': ['auto', 'sqrt'],
'min_samples_leaf': [2, 3, 5],
'min_samples_split': [2, 3, 5]
}

rf_clf = GridSearchCV(estimator=rf,
param_grid=param_grid,
scoring=scoring_metric,
cv=cv,
verbose=False,
n_jobs=-1)

best_rf_clf = rf_clf.fit(X_train, y_train)


I get the following confusion matric and learning curves (based on this link):

Notice that the cross-validation curve exceeds 90% when training includes all samples. However, when I got to the (pre-processed, imbalanced) test set, I get a significantly lower recall score around 79%.

y_pred = best_rf_clf.best_estimator_.fit(X_train, y_train).predict(X_test)

recall_score(y_test, y_pred)
# outputs 0.793


What can explain this difference? I have shuffled both sets to make sure nothing funny happens when splitting them. Apart from that, I cannot think of anything else. Thanks a lot!!