Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Question

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not?

Data science seems to seem class imbalance as problematic and needing special techniques to remedy this problem.

For instance, this DS.SE question takes it as self-evident that class imbalance is problematic, and no answers or comments push back on this notion; the answer to this DS.SE question believes class imbalance to be a problem in need of remedy; and this, this, and this advocate for data adjustments in order to solve what is taken as a self-evident problem.

Statisticians do not see class imbalance as a problem for supervised learning that requires special remedy. Multiple posts on Cross Validated (Statistics) Stack Exchange argue against this except for specific circumstances.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Vanderbilt's founder of the Department of Biostatistics, Frank Harrell, has at least two good blog posts and several tweets that argue against class imbalance being a problem, even if some of the arguments are indirect (especially in the blog).

Classification vs. Prediction

For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly.

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

As discusssed here, fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5.

"People are still pushing ridiculous methods like SMOTE."

"Don't mess with the data. Use the right stat methods."

"#MachineLearning advocates are amazingly still using SMOTE to ruin 'imbalanced' data before analysis and invalidate 'classifications' they develop."

(Notice that these criticisms of SMOTE are not specific to SMOTE and just want a better generator of synthetic data. The criticisms are of doing something to "ruin" the data, with SMOTE just being one way to "ruin" the natural class ratio.)

Why does data science see class imbalance as a problem when statistics says it is not? What do the statisticians miss about class imbalance that makes it problematic in data science?

An answer like, "You could get 99% accuracy just by classifying everything in the majority class," really speaks to why accuracy is a problematic measure of performance, which is most obvious in, but not exclusive to, situations with imbalance. Also, "Imbalance often leads to everything being classified as the majority category," is a criticism of the decision rule used on top of the raw predictions from most (but I concede not all) models, such as the probability values returned by logistic regressions (think model.predict_proba instead of model.predict in sklearn), which might be reasonable when they all fall below some software-default threshold and would be rounded to the majority category by some kind of predict method.

Answer 1

It's generally not related to Data Science but what goes around; typically all sort of bad practice relating to laziness / looking for short term rewards. I wouldn't say DS is pushing for it but rather avoiding to test it thoroughly trough robust tests.

Notably, the examples you provide are quite outdated and do not represent the whole Data Science approach, but some people make quick answer to get short term rewards. I can honestly provide you as much anecdotal evidence about some Data Scientist not finding imbalance to be a problem. One of my favorite would be this one: https://www.kaggle.com/competitions/amazon-employee-access-challenge/discussion/5086

I'd argue you cannot really make generalities about DS making a topic out of imbalance like that. The main problem is that there are significant incentives to create a problem when there is none and provide solutions. From my experience, once you try to test it robustly the problem mostly disappear. Over time I have found some use-cases where it mattered:

• Very strong imbalance means you have a lot of unusefull data. And the bottleneck of your pipeline is the maximum memory used. So under sampling the majority class is a good start to reduce memory usage without loss in performance.
• For NNs, it seems that some standard library do not accept batches without any positive instances. That is, for some NN implementations, you need to ensure a minimal balance so each batch has at least one positive instance.
• It seems that, for NN, having balanced batch accelerate learning: http://proceedings.mlr.press/v97/byrd19a.html but notice that balancing is done through weight rebalancing.

Also some evidence on why rebalancing would work:

• Some papers indicates a performance gain, but are quite rare/outdated and difficult to replicate in my opinion. I've never seen a replicable example of rebalancing significantly increasing performance.
• Adding noise to the data can make the model more robust (Gaussian noise is widely accepted). Generative techniques might work in that sense, but it is not exactly about rebalancing
• Measurable costs imbalance. Yes, nowadays the way to go would be to add weights or implement a custom loss. But I suspect that those approach were not always available, and possibly not the easiest way to go.
• I also suspect a confusion between cost imbalance and data imbalance (typically as you do) to play a role. This happen when cost imbalance is not measurable, but target imbalance is, or in some case cost imbalance quite match target imbalance, typically in finance.

Overall, that stuff relates to MLE aspects of the project, or confusion about business constraints, rather than the DS one. The over-focusing of DS on data imbalance is mostly due to confusion and is partially irrational and lead by some bad habits I describe below, in a rework of a previous answer on SMOTE (Why SMOTE is not used in prize-winning Kaggle solutions?) to tell in my opinion Why is class imbalance seen as a problem and why solution to it are popular:

• Bad research practices / work ethics (a.k.a. 'publish or perish') that lead to create solution to problem that marginally or do not exist, marginally new techniques, technique with non-robust increment in performance ... etc. despite any application on any real life data set. It's quite difficult in the middle of your PHd thesis / paper to just go 'well nevermind this problem don't exist, let's just throw my last 6 month of work into the bin - and the funding with it'. Some of those practices can be found in business too. It also apply in business, where under a short time constraint you have to produce something, find an article on it on TDS and cc it into a notebook, then send it to management that can't read python code.

• Bad influencer practices: (you know those linkedin people sharing half baked Tds article), that are led by visibility instead of quality. Unfortunately, this sort of behavior is also present on other forums, like here or Kaggle forums (enven encouraged by the vote / medal system). Basically you are encouraged to see a problem and solve it. I'd even goes to add that it is often an UX problem. Typically discussion on imbalance are more prevalent on forums where there is no downvote buton, typically TDS.

• Bad interviewing practices: somehow Imbalance has become an interview question, it appears on interview question lists. Now you have both non-technical people asking about class imbalance and young DS learning to answer a list of stuff when they are asked about data imbalance. This is not a context where actually tried the stuff matters.

• Bad ML practices: Using wrong metrics and poorly designed cv it is very easy to gain performance as some over/undersampling techniques are leaky. It is then easier to publish / grab attention from your employer with a gap up in performance and put another coin in the hype machine.

• Bad evaluation practices: even with a good metric if you add a bit of noise in your data you have roughly 50% chance to improve the performance. It is often really easy (often incentivised) to be lead by non-robust evaluation, both in business and research. When you have a gain in performance do you go tell your boss or do you go 'well this might not be robust' better hire an intern and spend the next 6 months reworking the thing ?

• Bad default parameters, as mentionned, With some library, if the user is lazy he will tend to not change the default parameters too much. And as, you mention it the choice of a default 0.5 threshold in sklearn predict is absolutely awfull when there is imbalance. Sklearn default threshold is not the only thing promoting imbalance as a problem but it might be the worst.

Overall it is not that much of a problem when you stop listening to inexperienced data-scientist. Typically G. Lemaitre, author of imbalanced learn package shows how imabalance is not a problem and how rebalancing is not useful: Get the best from your scikit-learn classifier.

Answer 2

Statisticians typically focus on probabilistic classification. In the way they build their models they are interested in predicting $P(Y|X) = P(X|Y)P(Y)/P(X)$. Now we find :

1. The predicted probabilities are seriously harmed when altering P(Y) during training. The most extreme case is when you have no predictive variable and consider estimating P(Y) only. Should you fudge with the natural prevalences of the outcomes of Y? No! If you observed Y=1 in 2 percent of the cases your model should estimate $\hat{P}(Y=1)=0.02$. Done. If you had some predictive variables and wanted to do a binary classification based on $\hat{P}(Y=1|X)=0.02$ do threshold tuning ( don't stick with a threshold of 0.5 which seems to be good mainly for balanced datasets ...). Also note that probabilistic classification is best evaluated with scoring rules https://en.m.wikipedia.org/wiki/Scoring_rule and not with metrics for binarized events (see https://www.fharrell.com/post/class-damage/ ). Some scoring rules like the one related to cross entropy loss are also directly related to maximum likelihood estimation of a data generating process.

Data Scientists may be interested in other goals than probabilistic classification. See the following two points:

2. what is a valid reason for being interested in binary classification instead of predicting probabilities? Well sometimes you might have to estimate expected costs based on classification outcomes. If you have a very imbalanced cost for your false positives or false negatives you may be able to reduce the expected costs by trading off between one type of error and another one. You do not necessarily need a calibrated model then.
3. you might wanna do data augmentation for improved generalization (but you should still maintain the class frequencies if you look for a calibrated classifier). Data augmentation may be valid and a great idea depending on the model and depending on the fact if the augmentation adds new information by introduction of systematic real world variation

While the statistician task 1) is a correct perspective also points 2-3 are valid (and more present in data science). 2 is itself a valid problem and 3 is a valid approach for adding information to your model.

However sometimes people end up wrongly and blindly advocating for upsampling or down sampling (especially when you have a fixed threshold at 0.5). Note that up/down sampling is (more or less) equivalent to assigning class weights to training examples - this is obvious when considering gradients .

Point 2 can be expanded to other classification settings: in a typical data science project you might find a multi-class classification setting where simple 1 dimensional threshold tuning will not work anymore to improve the precision/recall tradeoff for all classes equally. (And you might have to make this tradeoff due to a high cost imbalance)

How can you solve these multi target optimizations from point 2? You could use oversampling/downsampling/weighting as an indirect way to find different Pareto optimal points. Given different fractions of targets in up- or downsampling you will achieve different values of precision and recall and will move on the Pareto front. (red Pareto front (bigger is better) with x axis e.g. recall and y axis precision. Since there is no total order (but only the Pareto order https://en.m.wikipedia.org/w/index.php?title=Pareto_order) you cannot decide which point on the Pareto front is better, image by Berklas under CC-by SA)

Ways to find a Pareto optimal solution while avoiding to fudge with the class frequencies by upsampling or downsampling:

• The draft https://arxiv.org/abs/2201.08528v3 ("To SMOTE or not to SMOTE?") shows that for binary classification threshold-tuning is equally good to up or down sampling in strong learners (XGboost etc) and maintains calibration.
• With an explicit cost assignment you can alternatively do cost-sensitive machine learning and get class assignments which are cost-optimal. This scalarization with a cost function is a more explicit way to deal with the fact that in a multi-objective optimization (detecting multiple classes optimally) there is multiple incomparable Pareto optimal solutions.

Update: The above was mainly written with probabilistic classification in mind. The same facts, however, also hold true for regression. A statistician tends to describe the data generating processes (e.g. using maximum likelihood estimation) while a data scientist may have other goals in mind. When describing a data generating process you may want to achieve e.g.

• calibration (in problems where a data scientist sees the classification task)
• unbiasedness for the conditional mean (in regression where you use the mean squared error/ Gaussian log likelihood) or a conditional quantile (in quantile regression). Still sometimes also statisticians are interested in biased estimators because they have a lower variance (making a bias-variance tradeoff)

Oversampling or downsampling may affect the conditional bias of a regression model (in a similar way like it affects calibration) and the variance of the model: typically increasing the bias (by upsampling) reduces the variance.

It may be desirable to modify the bias of your model due to asymmetric costs of overestimation or underestimation. In Cost-sensitive regression a Pareto optimal solution is found by assigning costs for overestimation and underestimation. You can alternatively move along the Pareto front by assigning weights/over- undersampling your training examples. However, this is very indirect and you probably will not know which tradeoff you are going to get before performing an experiment.

Summary:

When solving a multi-objective optimization problem 2), rather than doing oversampling/undersampling with unknown a-priori outcome, you might better do:

• Cost-sensitive learning
• Regularization (for the bias-variance tradeoff)
• Threshold-tuning
• ...

in order to achieve your pareto-optimal solution in more direct way.

If you have a probabilistic classification problem 1), then a statistician would also warn you from binarizing it and perform evaluation of the probability model with metrics like accuracy, precision, recall, etc (see Frank Harrells post).

PS:

You should not confuse downsampling with rejection sampling which checks for some pattern in your data. In another scenario (not mentioned above) your training data could be drawn not at random, but by a biased sampling strategy. Then rejecting some of these biased samples based on a characteristic of the data might seem like downsampling while in reality it is a form of rejection sampling just correcting the biased sampling