How to compensate for class imbalance in prediction model?

Question

I'm trying to run a prediction model on a customers' data set to predict the likelihood that a new customer would be interested in buying product X, offered by a company that sells products X,Y and Z. E.g. would this guy, non-customer, of this age and salary, be interested in product X?

To train the model, I have a basin of 100K company customers, of which only 5K bought product X - the remaining 95K bought other products. Any prediction model guesses 'nobody will buy product X' accepting those ~5% false negatives.

How can I compensate for this skewness of the data? i.e. 95% vs. 5%? Thanks

Answer 1

There is a lot of information and techniques for rare event or imbalanced classes. Sorry to post links but (https://stats.stackexchange.com/search?q=rare+event and https://stats.stackexchange.com/search?q=imbalanced) but I do not want to duplicate all of the other work.

In my experiences, often I left the data set alone. I usually got good results. If that is the true ratio, then my model should know that. Sometimes I downsampled the majority class. I played with SMOTE (https://arxiv.org/pdf/1106.1813.pdf) before as well.

For your questions above, yes - changing the data with weights or over/unsampling may bias the results. Need to check. If you downsample, make sure you are not tossing out signal from the majority class.

Answer 2

You can try to set different weights on each samples. For example, you can set 0.05 weight on those who bought other products, and 0.95 weight on those who bought product X.

In sklearn, there's a sample_weight parameter which provides this method. In R, checkout this guide.

If you're using XGBoost, here is a guide which shows how to deal with imbalanced class in XGBoost.

Answer 3

You've described a null model for a logistic regression, which in this case would predict P(will buy X)=0.05. Any other model that incorporates your covariates of age and salary, will do better. The fact that there are 19 zeroes for every 1 in your response shouldn't stop you trying to make a model that does significantly better than a null model.

So just put age and salary in as linear terms in the regression and go from there.