Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
In a simple binary classification problem, at what point does majority class to minority class become significant become significant? Intuitively, I would expect a 3:1 ratio to not be an issue, maybe not even a 10:1 ratio. But a 100:1 ratio certainly does require some action. What might be this cutoff?
As a followup, what might be come potential solutions beyond undersampling and oversampling?
There is no strict threshold at which a dataset is considered imbalanced. Accordingly, in Foundations of Imbalanced Learning Gary M. Weiss writes:
There is no agreement, or standard, concerning the exact degree of class imbalance required for a data set to be considered truly "imbalanced." But most practitioners would certainly agree that a data set where the most common class is less than twice as common as the rarest class would only be marginally unbalanced, that data sets with the imbalance ratio about 10:1 would be modestly imbalanced, and data sets with imbalance ratios above 1000:1 would be extremely unbalanced. But ultimately what we care about is how the imbalance impacts learning, and, in particular, the ability to learn the rare classes.
A pragmatic approach could be to fit your models on the imbalanced dataset and check if the imbalance leads to large differences in performance between the classes. But keep in mind that the ultimate goal is to minimize misclassification cost. So performance difference between classes are not necessarily a problem if classes have equal misclassification cost (usually for imbalanced datasets the underlying assumption is that misclassification cost are not equally distributed, i.e. misclassification cost for minority classes are assumed to be higher).
To handle imbalanced datasets sampling-based approaches are most common but they are not limited to under- and over-sampling. There are also hybrid (e.g. SMOTE+Tomek) and Ensemble-based methods (e.g. BalancedRandomForest).
Moreover, there are algorithmic approaches which include cost-sensitive learning (e.g. Weighted Random Forest) and skew-insensitive learning (e.g. Naïve Bayes).
Finally, you can adapt the performance metric to be used. AUROC and ROC Curves are commonly used for imbalanced datasets. However, they can be biased too. Which is why some authors suggest to use Precision-Recall-Curves and AUC-PR instead. Moreover, Precision, Recall and F1 score are common as well.
Note however, that simply using a different performance metric for model selection and model evaluation will not make your models optimize for these internally when being trained. Which is why the other approaches and especially sampling are so frequently used.
