Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
For handling an imbalanced dataset, we have a variety of techniques like adjusting class weights, oversampling, undersampling, SMOTE and its different variations (RCSMOTE, GSMOTE, DBSMOTE).
My question is how can I identify the most optimal technique favorable for my model? Is it only by applying each of them and see which one gives the best metrics?
Are there any conditions/scenarios where a given technique is preferred over the other ones?
In almost all circumstances, sound statistics disputes that class imbalance is a problem. Consequently, you probably should not do anything besides keep doing good data science that is grounded in correct statistics. (After all, this is what you were doing the whole time, right?)
Imbalance often appears to be a problem because models can achieve high accuracy scores by always or almost always classifying as the majority class. The trouble with such a statement is that most models do not do classification. Despite what software methods might exist, models like logistic regressions and neural networks do not do classification. They output predictions on a continuum that can be binned into categories, but they do not have to be, and the way to separate the continuous predictions into discrete classes could involve a threshold other than the usual software-default of a probability of $0.5$ or even more discrete categories than there are classes.
By evaluating models on statistically sound values like log-loss (“crossentropy loss” or “negative log likelihood” in some circles) or Brier score, almost any apparent issue related to class imbalance turns out not to be a problem. Since class imbalance then is not a problem, there is no need to use methods like oversampling, undersampling, or synthesis of new points in order to solve a non-problem. In fact, since synthesis of new points need not produce reasonable new points, creating synthetic points might create problems just because the user tried to solve a non-problem.
We have an entire statistics meta post with oodles of good links related to class imbalance and why it is not the problem it appears to be.
Variance part:
Despite the somewhat popular notion, data augmentations won't magically fix the lack of data (see Is SMOTE any good at creating new points?). They could possibly decrease variance somewhat if a model is already doing well.
This becomes rather obvious if you try visualizing a bunch of rare observations and potential decision boundaries in a multidimensional feature space.
Bias part:
The exact method is not relevant as much as the proportion. Resampling violates the major concept of train set having the same distribution as the production one, thus giving your model a bias aimed to offset the bias of the logistic function. There's no rule of thumb for this as far as I'm aware, simply equalizing the populations is not guaranteed to perfectly offset such a bias. See Logistic Regression in Rare Events Data for more info.
TL;DR: Data augmentations aren't really there to combat imbalance. If you pursue the metric of certain class on a certain set, that's trial and error (computationally expensive, often yielding an insignificant improvement and prone to drifting in production).
If you need a statistically correct model, avoid resampling/weighting when doing augmentations and evaluate using metrics that operate on scores rather than predictions (and aren't class specific). The logistic bias can be offset by selecting the decision threshold, which is most often a single vectorized operation.
