I have a data set which has data on consumers and a flag for whether they have expressed interest in a product or not. I am looking to build a model using R which will be able to predict whether or not a consumer will be interested in the product, before they are shown the product. I will then be filtering based on this model output to reduce costs downstream by not showing the product to consumers whose data implies they wouldn't be interested.
Here is a summary of my data set:
number of rows: ~300k
number of columns: 40
column types: 1 bit (interest flag), 37 num, 2 char (categorical)
The char columns are employment/residential status and so contain things like full time, private tenant, etc.
Here is what I have tried so far:
- I have used the Boruta package in R for feature selection and then I have created a regression model using lm() and specifying all of the important features.
- I have used Stepwise regression for feature selection where I start with a model with 0 parameters and a model with all parameters and "step" between them until the best model is created. I then grab the important features and use lm() again to create a regression model.
The issue I am having is that because the interest flag is so rare (roughly 1,600 / 300,000 or 0.5%), the values that the model gives using predict() are significantly below the 0.5 threshold I've applied for the logistic regression model. This then manifests in the model basically saying no one is interested in the product. My questions are:
- Is this a result of the features I have not being descriptive enough to build a good model?
- What is the best way to build a model for such a rare event?