Logistic Regression to model a rare event

Question

I have a data set which has data on consumers and a flag for whether they have expressed interest in a product or not. I am looking to build a model using R which will be able to predict whether or not a consumer will be interested in the product, before they are shown the product. I will then be filtering based on this model output to reduce costs downstream by not showing the product to consumers whose data implies they wouldn't be interested.

Here is a summary of my data set:

• number of rows: ~300k

• number of columns: 40

• column types: 1 bit (interest flag), 37 num, 2 char (categorical)

The char columns are employment/residential status and so contain things like full time, private tenant, etc.

Here is what I have tried so far:

1. I have used the Boruta package in R for feature selection and then I have created a regression model using lm() and specifying all of the important features.
2. I have used Stepwise regression for feature selection where I start with a model with 0 parameters and a model with all parameters and "step" between them until the best model is created. I then grab the important features and use lm() again to create a regression model.

The issue I am having is that because the interest flag is so rare (roughly 1,600 / 300,000 or 0.5%), the values that the model gives using predict() are significantly below the 0.5 threshold I've applied for the logistic regression model. This then manifests in the model basically saying no one is interested in the product. My questions are:

1. Is this a result of the features I have not being descriptive enough to build a good model?
2. What is the best way to build a model for such a rare event?

Answer 1

Using lm is not the right approach to model a binary outcome. You would use a Logit in this case (see some example here and see why not lm here).

However, there are (at least) two more issues:

1. You have a highly unbalanced target
2. You may have "noisy" features

Regrading 1:

You should check if some oversampling of the minority class or using SMOTE would help to tackle the problem of the unbalanced target. Generally, when you have 99% of say class 1 (and 1% of class 0), you will often end up with an estimator with 99% accuracy while all predicted values are assigned to class 1. This of course is not a good prediction (since the "naive baseline" is 99% accuracy, namely assign all cases to class 1).

Regarding 2:

Logit may not be a good choice (depending on the nature of your explanatory variables). Logit is "linear" in parameters, meaning a logit model looks like $$ y = \beta_0 + \beta_1 x_1 + ... + \beta_n x_n +u. $$

In case the true data generating process is "more complex" you may benefit from using tree-based models which do not require any ex-ante parameterization such as Random Forest or Stochastic Gradient Boosting (e.g. xgboost). Make sure categorical features are encoded as "dummy" (or factor, depending on the model).

You could also check if feature generation helps to find more detailed aspects in the data. General idea: Generate features such as $x_1/x_2$, $x_1-x_2$ etc. and check via feature importance if they have good explanatory power. Keep the useful features on top of the features you would include anyway.

You should also check if there are features which are redundant. You could do this by looking at the feature importance from a Random Forest or from Boosting. Exclude features with little or no explanatory power.

You could also check if a Lasso/Ridge approach to (automatically) "shrink" redundant features would help. Lasso/Ridge can also be seen as a tool to investigate feature importance.

"Introduction to Statistical Learning" is a very good source to understand some of the models mentionned above (if you are not familiar with them). ISL also comes with R labs which should be helpful. The book can be downloaded for free.