Skip to main content
Data Science

Return to Question

Notice removed Canonical answer required by Dave
Bounty Ended with Lucas Morin's answer chosen by Dave
Notice added Canonical answer required by Dave
Bounty Started worth 500 reputation by Dave
Became Hot Network Question
added 997 characters in body
Source Link
Dave
Dave
  • 4.2k
  • 1
  • 9
  • 32

For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly.

As discusssed here, fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5.

"SMOTE is needed only because of the use of an improper discontinuous accuracy scoring rule. Don't"Don't mess with the data. Use the right stat methods."

"SMOTE is needed only because of the use of an improper discontinuous accuracy scoring rule. Don't mess with the data. Use the right stat methods."

For this reason the odd practice of subsampling the controls is used in an attempt to balance the frequencies and get some variation that will lead to sensible looking classifiers (users of regression models would never exclude good data to get an answer). Then they have to, in some ill-defined way, construct the classifier to make up for biasing the sample. It is simply the case that a classifier trained to a 1/2 prevalence situation will not be applicable to a population with a 1/1000 prevalence. The classifier would have to be re-trained on the new sample, and the patterns detected may change greatly.

As discusssed here, fans of “classifiers” sometimes subsample from observations in the most frequent outcome category (here Y=1) to get an artificial 50/50 balance of Y=0 and Y=1 when developing their classifier. Fans of such deficient notions of accuracy fail to realize that their classifier will not apply to a population when a much different prevalence of Y=1 than 0.5.

"Don't mess with the data. Use the right stat methods."

Source Link
Dave
Dave
  • 4.2k
  • 1
  • 9
  • 32

Why does data science see class imbalance as a problem for supervised learning when statistics does not?

Why does data science see class imbalance as a problem in supervised learning when statistics says it is not?

Data science seems to seem class imbalance as problematic and needing special techniques to remedy this problem.

For instance, this DS.SE question takes it as self-evident that class imbalance is problematic, and no answers or comments push back on this notion; the answer to this DS.SE question believes class imbalance to be a problem in need of remedy; and this, this, and this advocate for data adjustments in order to solve what is taken as a self-evident problem.

Statisticians do not see class imbalance as a problem for supervised learning that requires special remedy. Multiple posts on Cross Validated (Statistics) Stack Exchange argue against this except for specific circumstances.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help?

Profusion of threads on imbalanced data - can we merge/deem canonical any?

Vanderbilt's founder of the Department of Biostatistics, Frank Harrell, has at least two good blog posts and several tweets that argue against class imbalance being a problem, even if some of the arguments are indirect (especially in the blog).

Classification vs. Prediction

Damage Caused by Classification Accuracy and Other Discontinuous Improper Accuracy Scoring Rules

"People are still pushing ridiculous methods like SMOTE."

"SMOTE is needed only because of the use of an improper discontinuous accuracy scoring rule. Don't mess with the data. Use the right stat methods."

"#MachineLearning advocates are amazingly still using SMOTE to ruin 'imbalanced' data before analysis and invalidate 'classifications' they develop."

(Notice that these criticisms of SMOTE are not specific to SMOTE and just want a better generator of synthetic data. The criticisms are of doing something to "ruin" the data, with SMOTE just being one way to "ruin" the natural class ratio.)

Why does data science see class imbalance as a problem when statistics says it is not? What do the statisticians miss about class imbalance that makes it problematic in data science?

An answer like, "You could get 99% accuracy just by classifying everything in the majority class," really speaks to why accuracy is a problematic measure of performance, which is most obvious in, but not exclusive to, situations with imbalance. Also, "Imbalance often leads to everything being classified as the majority category," is a criticism of the decision rule used on top of the raw predictions from most (but I concede not all) models, such as the probability values returned by logistic regressions (think model.predict_proba instead of model.predict in sklearn), which might be reasonable when they all fall below some software-default threshold and would be rounded to the majority category by some kind of predict method.