Class imbalance strategies

Question

When dealing with the class imbalance problem in a binary classifier, there are three ways I know of to address it: over-sampling, under-sampling and using cost-sensitive methods.

Are there any guidelines, rules of thumb or general strategies to choose among these methods? A possible answer would be: over-sample when the positive class has more than 100 instances (I just made it up).

Answer 1

It's hard to put a general rule with this kind of methods, they depend heavily on the data at hand. You should know pros and cons of each, try all methods and see which performs best on the validation set. Remeber that:

• over-sampling the less frequent class can bring a lot of repetition in the data, because you just replicate the observations
• under-sampling can be bad because you throw away information in the most frequent class, and so you can lose performance

Hybrid methods tend to outperform both, see SMOTE or ROSE.

For example, SMOTE generates new information from the minority class (synthetic observation which are derived from the true samples), and also under-samples the majority class (you choose the final ratio of the two).

Again you need to test on the data the best solution.