14
$\begingroup$

Synthetic Minority Over-sampling Technique SMOTE, is a well known method to tackle imbalanced datasets. There are many papers with a lot of citations out-there claiming that it is used to boost accuracy in unbalanced data scenarios.

But then, when I see Kaggle competitions, it is rarely used, to the best of my knowledge there are no prize-winning Kaggle/ML competitions where it is used to achieve the best solution. Why SMOTE is not used in Kaggle?

I even see applied research papers (where there are millions of $ at stake) that SMOTE is not used: Practical Lessons from Predicting Clicks on Ads at Facebook

Is this because it's not the best strategy possible? Is it a research niche with no optimal real-life application? Is there any ML competition with a high-reward where this was used to achieve the best solution?

I guess I am just hesitant that creating synthetic data actually helps.

$\endgroup$
2

4 Answers 4

14
$\begingroup$

After some debate on social networks and much asking, see the Twitter thread.

The best answer that I can find is that it does not work.

I would love to retract my answer and see a real-life example where it actually works (see tweet JFPuget ).

A recap of other sources and social media:

(UPDATE) I found a paper does studies this problem "To SMOTE or not to SMOTE" https://arxiv.org/abs/2201.08528

$\endgroup$
3
7
$\begingroup$

Somehow I missed this question... I can answer this as I have spent a great part of the last years trying to deal with imbalance inside and outside of Kaggle.

Why SMOTE doesn't work?

SMOTE creates syntectic data points, which is roughly equivalent to oversampling (if you take points A and B, create (A+B)/2, it is roughly equivalent to weighting A and B by 1.5). The added complexity of the generation process doesn't seems to bring consistent performance.

I even think there might be some sort of 'no free lunch theorem' here: as you don't add or remove information, sometimes the new points are helpful, sometimes they are counter-productive, and the average impact should be 0.

Now the question is: does oversampling work ? Well it seems we have a decent answer over here: https://stats.stackexchange.com/questions/357466/are-unbalanced-datasets-problematic-and-how-does-oversampling-purport-to-he (roughly: No). I think that generally answer the question of the usefullness of SMOTE.

There are some times where undersampling/oversampling is actually needed for engineering constraints (undersampling reduces data size, and hence memory usage; oversampling ensures you have positives exemples in each mini-batches for NN, and it seems to accelerate convergence)

Why is it popular despite not being used on Kaggle?

Regarding Kaggle, I remember reading a winning solution mentioning SMOTE and that it only has marginal effect. I can't manage to find the discussion post. It might have been a minor / custom comp. I recently checked the blog posts and there is indeed no mention of SMOTE. But, why has it gained so much traction despite not being effective on Kaggle ?

  • Bad research practices (a.k.a. 'publish or perish') that only lead to diffusion of apparent increment on this solution, marginally new techniques, non-robust increment in performance ... etc. despite any application on any real life data set. Some of those practices can be found in business too.
  • Bad influencer practices (you know those linkedin people sharing half baked Tds article), that are led by visibility instead of quality. Unfortunately, this sort of behavior is also present on Kaggle forums (enven encouraged by the medal system).
  • Bad interviewing practices: somehow it has become an interview question, it appears on interview question lists. Now you have both non-technical people asking about it and young DS learning to answer 'SMOTE' when they are asked about data imbalance. A context were truth matters less than confidence.
  • Bad ML practices: Using wrong metrics and poorly designed cv it is very easy to gain performance as SMOTE is very leaky (it get easier to predict B from A when you add (A+B)/2 in your fold). It is then easier to publish with a gap up in performance and put another coin in the hype machine.
  • Bad evaluation practice: even with a good metric if you add a bit of noise in your data you have roughly 50% chance to improve the performance. It is often really easy (often incentivised) to be lead by non-robust evaluation, both inside and outside of Kaggle (business and research). It is quite rare to see discussion of robust evaluation of performance increment, even on kaggle.

Last but not least: Kagglers know a way better alternative. It should not be a surprise but Kaggle is very into gbdt that appears to be quite effective at handling imbalance. Notably each leaf of each tree being a set of exemple, you usually have a good probability calibration. A little bit of regularisation and you are good to go.

One major caveat to all of this: tabular comp. seems to get rarer over time (outside of TPS), and (maybe because they figured it is not a problem) there wasn't a significant imbalanced one in a while.

Cherry on top: you might want to see this Euroscipy Video from last week. G. Lemaitre from sklearn shows how to use it on an imbalanced dataset. It includes a demonstration of how SMOTE doesn't work. He also mention the package Imbalanced-Learn, that, as one of the creators he advise not to use (putting ADASYN and Tomek links in the same garbage bin). He also ends up talking about the real problem: cost imbalance matters more for business than target imbalance.

$\endgroup$
1
  • 1
    $\begingroup$ This is a good answer, thanks $\endgroup$ Commented Sep 2, 2023 at 6:20
2
$\begingroup$

SMOTE improves prediction performance in a few limited situations. However, generally, it does not work. See https://arxiv.org/abs/2201.08528

$\endgroup$
0
$\begingroup$

SMOTE or data augmentation techniques introduce bias.

In academic settings, researchers tend to overlook this, but in real world applications the issue is serious.

For example, some well-known companies used to offer AI services to law enforcement agencies that predicted where crime could happen. In actuality, the AI system simply sent and kept sending LE to low income areas where minority ethnic groups were the majority.

$\endgroup$
2
  • 1
    $\begingroup$ Your linked article doesn't seem to blame the model's badness on SMOTE. How does SMOTE specifically introduce bias? $\endgroup$
    – Ben Reiniger
    Commented Nov 10, 2023 at 16:54
  • 1
    $\begingroup$ @BenReiniger it fabricates (based on statistical or probabilistic measures) observations that never occured in the natural world. Recall, in engineering and computer science school we are reminded about the linearity of academic setting (assumptions) and the unpredictability of the natural world. The shared article was intended to introduce real world examples without delving deeper into the behind the scenes of the AI models, which we do not have access to because those are company secrets. $\endgroup$
    – Full Array
    Commented Nov 10, 2023 at 16:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.