Regression vs Classification to rank by profit

Question

Say that I have a population of 10k customers, for which I expect 100 responders in my next campaign and have budget to send a letter to 5k.

I have past data of how much those who respond spent, and I'm interested in getting the most $ out of my 5k letters (i.e. ideally I find the 100 responders, but if not, I want to find those who leave more money).

I have created a regression model and a classification model (1 if they spent money -i.e. reponded in the past - and 0 otherwise). After running bot models, I find that in a test sample of 10k, if I select the first 5k ranked by probability (for the classification model) and by the expected amount in the regression, the regression model performs better (not evaluating how many responders they found, but how much they found out of those selected).

Does it make sense based on the numbers of my context? I had expected the classification model to perform better, but can't figure out why the regression model is coming on top in my example.

Thank you very much!!

Answer 1

This actually isn't too surprising given the context in my opinion. Classifying an observation as "1" = spent money in past and "0" = did not spend money in past causes you to lose information that in your case is important; that is, the amount of money that each person gave.

Imagine these two scenarios, for example. In one scenario, you send your letter to ten people and all ten people send back 100 dollars each each. Using a classification scheme, these ten people are all "1"'s. In the second scenario, you send your letter to ten people again, but now only one of ten respond. However, this one person sends back a cheque of 10,000 dollars. From a pure profit point of view, clearly the 2nd option is superior.

Ultimately, both your classification and regression models answer two separate questions. Your classification model simply predicts how likely someone is to spend any amount of money, (by how you have phrased the question, anyway) whether that is 10 dollars or 10,000,000 dollars there exists no distinguishing between the two with a classification model (as you phrased it). The regression model predicts the exact amount that each user is expected to spend which if I am reading your question correctly, is what you care about.

If you want to make the problems closer in scope, perhaps try defining in your classification model as "1" if person spends over some profitable amount of money, and "0" if they don't. While I don't agree very much with doing this (you lose information this way by "bucketing" a continuous variable, and also, why make the problem harder?) you might get "closer" results.

Answer 2

You discard information when you do this as a classification problem. You consider someone who spent a dollar to be the same as someone who spent a gazillion bazillion dollars, when the former person is much more like someone who spent nothing.

It matters how much the person spends, not just if they spend. A few big spenders might make up for some misclassifications of people as spenders. If five people spend a dollar, that’s not as good for you as four who spend nothing and a fifth who spends 50 bucks.