I am working as a data scientist for the past 2 years where I have worked on problems related to binary classification, revenue prediction etc.
In the past two years, I have had 2 problems that focused specifically on binary classification with imbalanced data and size of datasets were low. In my first project it was 2977 (77:23) records and 2nd project was 3400 (70:30) records.. though I feel it is not extremely imbalanced but still slight imbalance..
I tried all the approaches that I know to do a best job - Threshold moving, considering various metrics to assess the performance of a model holistically, extensive feature engineering approaches etc...
Despite all this, I could never make the minority class precision or recall touch even 70% in the validation data..
So, am not sure whether it is impossible to achieve decent performance, problem is not suitable for prediction on imbalanced datasets or it shows poor performance by a data scientist like me.
Whatever tutorial or articles that I read online for imbalanced datasets, also show similar stories only where the performance of minority class is only around 50-60%
Meaning, they show without SMOTE, Resampling etc and after applying SMOTE, resampling etc, the performance goes up by few points and reaches 55-63% (just by 2 to 3 points)
How do big corporations and hospitals that works on fraud analytics and death likelihood etc do differently to deploy such models in production?
Any experience here anyone? Do they also settle for low performance but still go ahead with it as something is better than nothing?
Is it even possible to achieve 90% and above for precision, recall and f1 of minority class (which is our class of interest).
Can any experts here share some of your views?
Ps - whatever model I built earned some revenue to company..but not sure whether it is biz demand or model working...company believes model helped...but due to poor metrics, I don't believe in it though I have been given credit