Method of accuracy improve for binary classification imbalanced dataset

Question

I have an imbalance data set where the imbalance ratio No: Yes is 8:1. If I run classifiers on the groundtruth dataset I got recall and F2 measure for Naive bayes, Logistic regression, random forest. The recall and F2 measure somewhere between 0.500 to 0.200. The AUROC measures varies between 0.860 to 0.799. However, if I apply SMOTE method and make the dataset almost balanced where the No: Yes class ratio 2:1. I got recall and F2 measure better than previous approach and the ratio varies 0.650 t0 0.350 but AUROC getting poor and varies between 0.800 to 0.600.

The dataset is a categorical dataset.

What does this results imply?

If I like to improve the performance of all metrics with compare to groundtruth dataset what should be my approach?

Answer 1

The fact that the AUC doesn't improve while recall and F2-score do suggests that the resampling ends up being more or less equivalent to modifying the probability threshold on the non-resampled model.

The first thing I would recommend is to understand how the performance changes depending on the threshold, based on the original training data. In particular you could pick the threshold which maximizes F2-score, for instance.

Beyond this, it's important to make sure that the model doesn't overfit by comparing training and test set performance. A good way to truly improve performance imho is to analyze and improve the features, if possible: why does the model confuse positive and negative cases? Is there any helpful information in the features that could be represented differently to contribute to the model?