Logistic regression on biased data

Question

I am currently working on a dataset to predict customer attrition based on past data and transactions of the customers.

There are 2,40,000 customers in total out of which around 1,77,000 customers are active(as of today) while the remaining ones are inactive (6300). This is how sample headers look like :

custID|custAge|custGender|TQuantity|TVolume|TValue|TAmount|HolidayStatus|...

Overall, I have 40 predictors which include customer details, transaction details, item details etc.

The data obviously has more active customers than inactive customers i.e. inactive customers form only 2.6% of the entire customer base. Due to this, there are more transactions conducted by active customers(25million/32million) than by inactive (previously active) (6million/32million) ones.

Despite this, I created a logistic regression model using random data (shuf -n 500000 data.csv). The model achieves 96.69% base accuracy in predicting when fed with random data.

The problem: How to make the model predict with greater accuracy on such a biased dataset? or How do I sample the data more appropriately?

Model prediction: With 99.7% probablity, it predicts that the customer will be active whereas the customer is inactive

PS: Changing threshold won't help much

Answer 1

Background

I'll start with some background to help you research the solution yourself and then will add some specifics. What you refer to as "biased data" is more commonly known as unbalanced classes in the data science world. Also "customer turnover" is often referred to as churn.

Metrics

As hoards of Ng'ian devotees will undoubtably point out you need to start by designing a set of metrics that work better with unbalanced classes than accuracy. Accuracy does a poor job in testing the quality of predictions for unbalanced classes e.g. a cancer test for a cancer that occurs in 0.05 % of the population is 99.95% accurate if it always predicts "no cancer". I suggest using the F1-score as the key metric in cross-validating your model. The F1-score is the harmonic mean of precision and recall and tends to work both for balanced and unbalanced classes. There are other rations of harmonic mean, that could work in special cases, so be aware of these.

There are other metrics you should learn about also. ROC-AUC is likely at the top of the list for other metrics you should understand and know about.

Model Selection and Cross-Validation

Beginning a classification task with Logistic Regression is a fantastic strategy. I make a point to always use a linear regression for regression tasks and a logistic regression for classification tasks. The linear model provides significant insight into the feature importance and helps frame the problem.

But following this initial survey you should move on to other, more sophisticated models. Many will give you a litany of things to try. You should perhaps focus on one or two and develop the model while paying very careful attention to bias and variance as you cross-validate and test your model. A full bias-variance decomposition may be unnecessary, once you develop better intuition, but is a great place for newbs to start.

I suggest starting with an SVM and also eventually trying a random forest or naive Bayes model as this will traverse several regimes of model types (analogy, decision trees, bagging, Bayesian).

Finally... Unbalanced Classes

There are two typical methods for dealing with unbalanced classes. These include oversampling the minority class, and fixing the model by altering the hyperplane (SVM) or changing priors (Bayes).

There are lots of summaries of this problem and solution if you search for "unbalanced classes". But, it can still be a tricky problem despite all the literature. Good luck...

Hope this helps!

Answer 2

While there are well known techniques like down sampling for dealing with imbalanced classes (this is common in the finance industry where bankruptcies occur around only 1-3% of the time), I think in your case the model type will be much more important. If you are prioritizing explanatory power over predictive power or if you have a small data set, then logistic regression is fine. However, if you have a large dataset (which you have) and if you care more about predicting (which it sounds like you do) then you should pick a more advanced type of model.

Some good examples: random forest, gradient boosting, and support vector machines. Tree based methods likes gradient boosting and random forests are able to identify variable interactions that you would have discover manually / yourself if you went with logistic regression. These models are easily accessible in R (caret), Python (scikit-learn), or Java/Scala (Spark's ML lib or Weka).