Are imbalanced data problems solvable? [closed]

Question

I am working as a data scientist for the past 2 years where I have worked on problems related to binary classification, revenue prediction etc.

In the past two years, I have had 2 problems that focused specifically on binary classification with imbalanced data and size of datasets were low. In my first project it was 2977 (77:23) records and 2nd project was 3400 (70:30) records.. though I feel it is not extremely imbalanced but still slight imbalance..

I tried all the approaches that I know to do a best job - Threshold moving, considering various metrics to assess the performance of a model holistically, extensive feature engineering approaches etc...

Despite all this, I could never make the minority class precision or recall touch even 70% in the validation data..

So, am not sure whether it is impossible to achieve decent performance, problem is not suitable for prediction on imbalanced datasets or it shows poor performance by a data scientist like me.

Whatever tutorial or articles that I read online for imbalanced datasets, also show similar stories only where the performance of minority class is only around 50-60%

Meaning, they show without SMOTE, Resampling etc and after applying SMOTE, resampling etc, the performance goes up by few points and reaches 55-63% (just by 2 to 3 points)

How do big corporations and hospitals that works on fraud analytics and death likelihood etc do differently to deploy such models in production?

Any experience here anyone? Do they also settle for low performance but still go ahead with it as something is better than nothing?

Is it even possible to achieve 90% and above for precision, recall and f1 of minority class (which is our class of interest).

Can any experts here share some of your views?

Ps - whatever model I built earned some revenue to company..but not sure whether it is biz demand or model working...company believes model helped...but due to poor metrics, I don't believe in it though I have been given credit

Answer 1

The issue is not with the imbalance per se. The issue is that your categories are not particularly separable on the available data (or they are but you are not modeling the correct relationship, e.g., needing a quadratic term yet lacking one).

When imbalanced categories are easy to distinguish, performance is high. For instance, I see a lot more Honda cars than Ferrari cars (imbalanced classes). Nonetheless, I do not strugle to distinguish between the two, beause they look so different.$^{\dagger}$ In this case, the class imbalance is not an issue, and it is easy to identify the correct car manufacturer just about every time. On the other hand, I see these two "identical" twins about equally often (no imbalance), and I struggle to tell them apart, since they look so similar. In this case, despite the lack of imbalance, I mix up their names all the time and struggle to distinguish between them.

Two Cross Validated links are worth reading.

How to know that your machine learning problem is hopeless? The gist here is that some problems are just hard, such as hoping to predict the toss of a fair coin.

Are unbalanced datasets problematic, and (how) does oversampling (purport to) help? The gist here is that imbalanced problems are not so inherently different from balanced problems.

$^{\dagger}$I am reminded of a quote from the movie My Cousin Vinny, which is a possible spoiler.

They are discussing if getaway vehicles could be mistaken for each other: "One was the Corvette, which could never be confused with the Buick Skylark."