1
$\begingroup$

I have a binary classification task related to customer churn for a bank. The dataset contains 10,000 instances and 11 features. The target variable is imbalanced (80% remained as customers (0), 20% churned (1)).

My approach is the following: I first split the dataset into training and test sets, while preserving the 80-20 ratio for the target variable in both sets. I keep 8,000 instances in the training set and 2,000 in the test set. After pre-processing, I address the class imbalance in the training set with SMOTEENN:

from imblearn.combine import SMOTEENN

smt = SMOTEENN(random_state=random_state)
X_train, y_train = smt.fit_sample(X_train, y_train)

Now, my training set has 4774 1s and 4182 0s. I know proceed to building ML models. I use scikit-learn’s GridSearchCV with cv = KFold(n_splits=5, shuffle=True, random_state=random_state) and optimise based on the recall score. For instance, for a Random Forest Classifier:

cv = KFold(n_splits=5, shuffle=True, random_state=random_state)
scoring_metric='recall'

rf = RandomForestClassifier(random_state=random_state)
param_grid = {
    'n_estimators': [100],
    'criterion': ['entropy', 'gini'],
    'bootstrap': [True, False],
    'max_depth': [6],
    'max_features': ['auto', 'sqrt'],
    'min_samples_leaf': [2, 3, 5],
    'min_samples_split': [2, 3, 5]
}

rf_clf = GridSearchCV(estimator=rf,
                      param_grid=param_grid,
                      scoring=scoring_metric,
                      cv=cv,
                      verbose=False,
                      n_jobs=-1)

best_rf_clf = rf_clf.fit(X_train, y_train)

I get the following confusion matric and learning curves (based on this link):

enter image description here

Notice that the cross-validation curve exceeds 90% when training includes all samples. However, when I got to the (pre-processed, imbalanced) test set, I get a significantly lower recall score around 79%.

y_pred = best_rf_clf.best_estimator_.fit(X_train, y_train).predict(X_test)

recall_score(y_test, y_pred)
# outputs 0.793

What can explain this difference? I have shuffled both sets to make sure nothing funny happens when splitting them. Apart from that, I cannot think of anything else. Thanks a lot!!

$\endgroup$
2
$\begingroup$

Training on the resampled data causes the model to assign roughly the same proportion of positive and negative labels, especially in difficult cases where it doesn't have clear enough indications from the features.

When testing on the test set with the original distribution (this is the correct method of course), the model applies what it learned: for "difficult cases", it assigns roughly half positive labels. But the test set has a higher proportion of positive cases, so this results in more false negative errors and therefore lower recall than on the training set.

$\endgroup$
1
  • $\begingroup$ Thanks a lot for the explanation, Erwan! $\endgroup$
    – KK_o7
    Nov 24 at 11:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.