Outlier detection for unbalanced classes

Question

I have to make a predictive model for predicting a boolean Won/Lost variable based on some other numeric data; and further find out the features of observations that have 'Won'.

However, the number of 'Won's' in my dataset is 0.05%. I've tried both oversampling and downsampling, but it hasn't worked. Even if I take an equal amount of 'Won's and 'Lost's', the model is not accurate for the rest of the 'Lost' values. I've also tried out weights, but it's not working well. Ideally I think I have to put a very high weight for 'Won'.

PS: Using RandomForestClassifier, with a confusion matrix to verify.

I'm not keen on trying out SMOTE, as I've heard it's tough in Python.

So now I'm trying to look at it in a different way, and do anomaly detection for the 'Won' case, as it natural for the data to have so few 'Won' cases. So, two questions

1. Is this a correct approach?
2. How to go about it using Python?

Answer 1

you need to distinguish between these cases:

1. Data Imbalance
2. Data Imbalance + Very few number of samples (minority class)
3. Severe Data Imbalance + Very few number of samples (minority class)

20:60 vs. 10:20 vs. 100:1000 vs. 10:100

and these cases:

1. similarities between different classes.

2. wide variations within the same class.

You need to understand to which of these cases your problem belong.

if you have very severe data imbalance + very few number of samples + wide variation within the majority class and similarities between different classes. regular oversampling or down sampling techniques will not help you as well as most of the synthetic oversampling techniques designed specifically to deal with the data imbalance but the assumption is to have enough number of samples.

Try to focus more on ensemble techniques that designed mainly to deal with data imbalance. SMOTE-Boost RUSBoost SMOTEBagging IIIVote EasyEnsemble

Answer 2

Firstly, anamoly detection does not look to be the right approach for your use case.

Approach 1: There is a python module to perform under and over sampling with various techniques here. This package includes SMOTE implementation as well. Using this package try to balance event data in your sample before running RandomForest or any other classification algorithms.

Approach 2: If the above does not work well, then firth's bias reduced logistic regression approach with penalized profile likelihood based confidence intervals for parameter estimates will suit your case. You can try L1 (Lasso) regularization on logistic regression from Python scikit library. Alternatively you can try R's 'logistf' package example.