15
$\begingroup$

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not?

Data science seems to seem class imbalance as problematic and needing special techniques to remedy this problem.

For instance, this DS.SE question takes it as self-evident that class imbalance is problematic, and no answers or comments push back on this notion; the answer to this DS.SE question believes class imbalance to be a problem in need of remedy; and this, this, and this advocate for data adjustments in order to solve what is taken as a self-evident problem.

Statisticians do not see class imbalance as a problem for supervised learning that requires special remedy. Multiple posts on Cross Validated (Statistics) Stack Exchange argue against this except for specific circumstances.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Vanderbilt's founder of the Department of Biostatistics, Frank Harrell, has at least two good blog posts and several tweets that argue against class imbalance being a problem, even if some of the arguments are indirect (especially in the blog).

Classification vs. Prediction

For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly.

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

As discusssed here, fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5.

"People are still pushing ridiculous methods like SMOTE."

"Don't mess with the data. Use the right stat methods."

"#MachineLearning advocates are amazingly still using SMOTE to ruin 'imbalanced' data before analysis and invalidate 'classifications' they develop."

(Notice that these criticisms of SMOTE are not specific to SMOTE and just want a better generator of synthetic data. The criticisms are of doing something to "ruin" the data, with SMOTE just being one way to "ruin" the natural class ratio.)

Why does data science see class imbalance as a problem when statistics says it is not? What do the statisticians miss about class imbalance that makes it problematic in data science?

An answer like, "You could get 99% accuracy just by classifying everything in the majority class," really speaks to why accuracy is a problematic measure of performance, which is most obvious in, but not exclusive to, situations with imbalance. Also, "Imbalance often leads to everything being classified as the majority category," is a criticism of the decision rule used on top of the raw predictions from most (but I concede not all) models, such as the probability values returned by logistic regressions (think model.predict_proba instead of model.predict in sklearn), which might be reasonable when they all fall below some software-default threshold and would be rounded to the majority category by some kind of predict method.

$\endgroup$
11
  • 1
    $\begingroup$ @noe I've edited to include quotes. $\endgroup$
    – Dave
    Commented Jan 9 at 19:46
  • 3
    $\begingroup$ I think DS is mostly wrong about this. And it is mostly due to behavioral aspects. See my answer here: datascience.stackexchange.com/questions/106461/… there are a lot of ways in which data imbalance / oersampling techniques like SMOTE survived. $\endgroup$ Commented Jan 9 at 21:53
  • 1
    $\begingroup$ LLMs are notorious for botching their responses to technical questions. $\endgroup$
    – Dave
    Commented Jan 10 at 23:09
  • 1
    $\begingroup$ LLMs are largely trained on internet corpora, which are filled with blog posts for beginner data scientists that discuss the "problem" of imbalanced data. $\endgroup$
    – Ben Reiniger
    Commented Jan 11 at 17:06
  • 1
    $\begingroup$ @BenReiniger a fair few of those blog posts are also written by beginner data scientists, so there is a positive feedback loop! $\endgroup$ Commented May 30 at 11:05

2 Answers 2

9
+500
$\begingroup$

It's generally not related to Data Science but what goes around; typically all sort of bad practice relating to laziness / looking for short term rewards. I wouldn't say DS is pushing for it but rather avoiding to test it thoroughly trough robust tests.

Notably, the examples you provide are quite outdated and do not represent the whole Data Science approach, but some people make quick answer to get short term rewards. I can honestly provide you as much anecdotal evidence about some Data Scientist not finding imbalance to be a problem. One of my favorite would be this one: https://www.kaggle.com/competitions/amazon-employee-access-challenge/discussion/5086

I'd argue you cannot really make generalities about DS making a topic out of imbalance like that. The main problem is that there are significant incentives to create a problem when there is none and provide solutions. From my experience, once you try to test it robustly the problem mostly disappear. Over time I have found some use-cases where it mattered:

  • Very strong imbalance means you have a lot of unusefull data. And the bottleneck of your pipeline is the maximum memory used. So under sampling the majority class is a good start to reduce memory usage without loss in performance.
  • For NNs, it seems that some standard library do not accept batches without any positive instances. That is, for some NN implementations, you need to ensure a minimal balance so each batch has at least one positive instance.
  • It seems that, for NN, having balanced batch accelerate learning: http://proceedings.mlr.press/v97/byrd19a.html but notice that balancing is done through weight rebalancing.

Also some evidence on why rebalancing would work:

  • Some papers indicates a performance gain, but are quite rare/outdated and difficult to replicate in my opinion. I've never seen a replicable example of rebalancing significantly increasing performance.
  • Adding noise to the data can make the model more robust (Gaussian noise is widely accepted). Generative techniques might work in that sense, but it is not exactly about rebalancing
  • Measurable costs imbalance. Yes, nowadays the way to go would be to add weights or implement a custom loss. But I suspect that those approach were not always available, and possibly not the easiest way to go.
  • I also suspect a confusion between cost imbalance and data imbalance (typically as you do) to play a role. This happen when cost imbalance is not measurable, but target imbalance is, or in some case cost imbalance quite match target imbalance, typically in finance.

Overall, that stuff relates to MLE aspects of the project, or confusion about business constraints, rather than the DS one. The over-focusing of DS on data imbalance is mostly due to confusion and is partially irrational and lead by some bad habits I describe below, in a rework of a previous answer on SMOTE (Why SMOTE is not used in prize-winning Kaggle solutions?) to tell in my opinion Why is class imbalance seen as a problem and why solution to it are popular:

  • Bad research practices / work ethics (a.k.a. 'publish or perish') that lead to create solution to problem that marginally or do not exist, marginally new techniques, technique with non-robust increment in performance ... etc. despite any application on any real life data set. It's quite difficult in the middle of your PHd thesis / paper to just go 'well nevermind this problem don't exist, let's just throw my last 6 month of work into the bin - and the funding with it'. Some of those practices can be found in business too. It also apply in business, where under a short time constraint you have to produce something, find an article on it on TDS and cc it into a notebook, then send it to management that can't read python code.

  • Bad influencer practices: (you know those linkedin people sharing half baked Tds article), that are led by visibility instead of quality. Unfortunately, this sort of behavior is also present on other forums, like here or Kaggle forums (enven encouraged by the vote / medal system). Basically you are encouraged to see a problem and solve it. I'd even goes to add that it is often an UX problem. Typically discussion on imbalance are more prevalent on forums where there is no downvote buton, typically TDS.

  • Bad interviewing practices: somehow Imbalance has become an interview question, it appears on interview question lists. Now you have both non-technical people asking about class imbalance and young DS learning to answer a list of stuff when they are asked about data imbalance. This is not a context where actually tried the stuff matters.

  • Bad ML practices: Using wrong metrics and poorly designed cv it is very easy to gain performance as some over/undersampling techniques are leaky. It is then easier to publish / grab attention from your employer with a gap up in performance and put another coin in the hype machine.

  • Bad evaluation practices: even with a good metric if you add a bit of noise in your data you have roughly 50% chance to improve the performance. It is often really easy (often incentivised) to be lead by non-robust evaluation, both in business and research. When you have a gain in performance do you go tell your boss or do you go 'well this might not be robust' better hire an intern and spend the next 6 months reworking the thing ?

  • Bad default parameters, as mentionned, With some library, if the user is lazy he will tend to not change the default parameters too much. And as, you mention it the choice of a default 0.5 threshold in sklearn predict is absolutely awfull when there is imbalance. Sklearn default threshold is not the only thing promoting imbalance as a problem but it might be the worst.

Overall it is not that much of a problem when you stop listening to inexperienced data-scientist. Typically G. Lemaitre, author of imbalanced learn package shows how imabalance is not a problem and how rebalancing is not useful: Get the best from your scikit-learn classifier.

$\endgroup$
6
  • 1
    $\begingroup$ I have added some cases where it mattered, relating mostly to Engineering aspects of the DS pipeline, not the overall performance of the model. I am afraid that this imbalance craze is mostly irrational, not driven by robustly observed performance gain. $\endgroup$ Commented Jan 17 at 20:15
  • $\begingroup$ ...in some case cost imbalance quite match target imbalance, typically in finance. Could you please expand on why you call out finance in particular? $\endgroup$
    – Dave
    Commented Jan 24 at 21:15
  • $\begingroup$ Note the description of G Lemaitre in the video is correctly describing that up- or downsampling do not surpassing the benefits of threshold-tuning to achieve some (cost) metric. However it falls short in describing alternatives to threshold tuning in order to achieve your multiobjective target.** Scalarization of multiple objectives by using a cost metric and then performing threshold tuning is just one of many approaches **! See my post. $\endgroup$
    – Ggjj11
    Commented Jan 25 at 12:17
  • $\begingroup$ Yeah that's why I describe them in my post. The general problem is cost imbalance, that you better take in your loss function... the thresholding (need for binarisation) is an entirely different topic $\endgroup$ Commented Jan 25 at 12:35
  • $\begingroup$ @Dave because I am familiar with it, I can't really speak of other topics. In finance (credit) you have a direct link, or rather a 'profitability' inequation. Basically if you charge a constant interest rate (cost imbalance), it needs to be above the default rate (target imbalance) to be profitable. It is often taken as an equality, which is mostly true in a competitive market. $\endgroup$ Commented Jan 25 at 12:40
2
$\begingroup$

Statisticians typically focus on probabilistic classification. In the way they build their models they are interested in predicting $P(Y|X) = P(X|Y)P(Y)/P(X)$. Now we find :

  1. The predicted probabilities are seriously harmed when altering P(Y) during training. The most extreme case is when you have no predictive variable and consider estimating P(Y) only. Should you fudge with the natural prevalences of the outcomes of Y? No! If you observed Y=1 in 2 percent of the cases your model should estimate $\hat{P}(Y=1)=0.02$. Done. If you had some predictive variables and wanted to do a binary classification based on $\hat{P}(Y=1|X)=0.02$ do threshold tuning ( don't stick with a threshold of 0.5 which seems to be good mainly for balanced datasets ...). Also note that probabilistic classification is best evaluated with scoring rules https://en.m.wikipedia.org/wiki/Scoring_rule and not with metrics for binarized events (see https://www.fharrell.com/post/class-damage/ ). Some scoring rules like the one related to cross entropy loss are also directly related to maximum likelihood estimation of a data generating process.

Data Scientists may be interested in other goals than probabilistic classification. See the following two points:

  1. what is a valid reason for being interested in binary classification instead of predicting probabilities? Well sometimes you might have to estimate expected costs based on classification outcomes. If you have a very imbalanced cost for your false positives or false negatives you may be able to reduce the expected costs by trading off between one type of error and another one. You do not necessarily need a calibrated model then.
  2. you might wanna do data augmentation for improved generalization (but you should still maintain the class frequencies if you look for a calibrated classifier). Data augmentation may be valid and a great idea depending on the model and depending on the fact if the augmentation adds new information by introduction of systematic real world variation

While the statistician task 1) is a correct perspective also points 2-3 are valid (and more present in data science). 2 is itself a valid problem and 3 is a valid approach for adding information to your model.

However sometimes people end up wrongly and blindly advocating for upsampling or down sampling (especially when you have a fixed threshold at 0.5). Note that up/down sampling is (more or less) equivalent to assigning class weights to training examples - this is obvious when considering gradients .

Point 2 can be expanded to other classification settings: in a typical data science project you might find a multi-class classification setting where simple 1 dimensional threshold tuning will not work anymore to improve the precision/recall tradeoff for all classes equally. (And you might have to make this tradeoff due to a high cost imbalance)

How can you solve these multi target optimizations from point 2? You could use oversampling/downsampling/weighting as an indirect way to find different Pareto optimal points. Given different fractions of targets in up- or downsampling you will achieve different values of precision and recall and will move on the Pareto front.red Pareto front (bigger is better) with x axis e.g. recall and y axis precision. Since there is no total order (but only the Pareto order https://en.m.wikipedia.org/w/index.php?title=Pareto_order) you cannot decide which point on the Pareto front is better, image by Berklas under CC-by SA (red Pareto front (bigger is better) with x axis e.g. recall and y axis precision. Since there is no total order (but only the Pareto order https://en.m.wikipedia.org/w/index.php?title=Pareto_order) you cannot decide which point on the Pareto front is better, image by Berklas under CC-by SA)

Ways to find a Pareto optimal solution while avoiding to fudge with the class frequencies by upsampling or downsampling:

  • The draft https://arxiv.org/abs/2201.08528v3 ("To SMOTE or not to SMOTE?") shows that for binary classification threshold-tuning is equally good to up or down sampling in strong learners (XGboost etc) and maintains calibration.
  • With an explicit cost assignment you can alternatively do cost-sensitive machine learning and get class assignments which are cost-optimal. This scalarization with a cost function is a more explicit way to deal with the fact that in a multi-objective optimization (detecting multiple classes optimally) there is multiple incomparable Pareto optimal solutions.

Update: The above was mainly written with probabilistic classification in mind. The same facts, however, also hold true for regression. A statistician tends to describe the data generating processes (e.g. using maximum likelihood estimation) while a data scientist may have other goals in mind. When describing a data generating process you may want to achieve e.g.

  • calibration (in problems where a data scientist sees the classification task)
  • unbiasedness for the conditional mean (in regression where you use the mean squared error/ Gaussian log likelihood) or a conditional quantile (in quantile regression). Still sometimes also statisticians are interested in biased estimators because they have a lower variance (making a bias-variance tradeoff)

Oversampling or downsampling may affect the conditional bias of a regression model (in a similar way like it affects calibration) and the variance of the model: typically increasing the bias (by upsampling) reduces the variance.

It may be desirable to modify the bias of your model due to asymmetric costs of overestimation or underestimation. In Cost-sensitive regression a Pareto optimal solution is found by assigning costs for overestimation and underestimation. You can alternatively move along the Pareto front by assigning weights/over- undersampling your training examples. However, this is very indirect and you probably will not know which tradeoff you are going to get before performing an experiment.

Summary:

When solving a multi-objective optimization problem 2), rather than doing oversampling/undersampling with unknown a-priori outcome, you might better do:

  • Cost-sensitive learning
  • Regularization (for the bias-variance tradeoff)
  • Threshold-tuning
  • ...

in order to achieve your pareto-optimal solution in more direct way.

If you have a probabilistic classification problem 1), then a statistician would also warn you from binarizing it and perform evaluation of the probability model with metrics like accuracy, precision, recall, etc (see Frank Harrells post).

PS:

You should not confuse downsampling with rejection sampling which checks for some pattern in your data. In another scenario (not mentioned above) your training data could be drawn not at random, but by a biased sampling strategy. Then rejecting some of these biased samples based on a characteristic of the data might seem like downsampling while in reality it is a form of rejection sampling just correcting the biased sampling

$\endgroup$
7
  • $\begingroup$ This notion of finding Pareto-optimal points seems like it should be the focus of the answer, like it might be a legitimate reason to want to remedy class imbalance. $\endgroup$
    – Dave
    Commented Jan 18 at 19:18
  • $\begingroup$ This is still so rambling that I cannot tell what point you're trying to make, and I really want to understand your argument about Pareto optimality (whether that argument is correct or incorrect). $\endgroup$
    – Dave
    Commented Jan 20 at 22:47
  • $\begingroup$ @Dave I added more structure $\endgroup$
    – Ggjj11
    Commented Jan 21 at 10:59
  • $\begingroup$ I tell you that statisticians try to achieve a calibrated probability estimate and that upsampling/downsampling is not an option then. I also tell you that in the case with multi-objective optimization or cost-sensitive learning in a data science use-case there can be valid application for upsampling or downsampling. However, you should consider the alternative, more direct options to find a Pareto optimum. $\endgroup$
    – Ggjj11
    Commented Jan 21 at 11:58
  • 1
    $\begingroup$ Where do you stop following exactly? What are the questions which you have? You need to see the full picture. There is cases where you prefer calibration (typically statisticians have these cases in mind) - then upsampling/downsampling is bad. There is other cases where you need to perform a pareto optimization with multiple targets and you do not care about calibration too much - then upsampling/downsampling is one of many options. The post above outlines all of these distinct cases under different scenarios. This is why it appears bumpy - but it isn't there is not ONE story line... $\endgroup$
    – Ggjj11
    Commented Jan 22 at 18:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.