# Modeling when the response variable has too many 0's and few continuous values?

For problems where the data represents online fraud or insurance (where each row represents a transaction), it is typical for the response variable to denote the value of fraud committed in dollars. Such a response value might have less than 5% non-zero values denoting fraudulent transactions.

I have two questions regarding such a dataset:

1. What algorithms can we use to ensure that the model not only predicts the fraudulent transactions accurately, but also predicts the value of fraud associated with these.
2. Assuming that we can quantify the cost involved in each false positive (tagging a non-fraudulent transaction as fraudulent) and cost incurred due to a false negative (tagging a fraudulent transaction as non-fraudulent), how can we optimize the model to maximize savings (or minimize losses)?
• You'll find lots of hits if you search stats.SE for "imbalanced" data. – Emre Nov 12 '14 at 18:13
• @Emre: This question is focused more on the solution to fraud like problems rather than handling imbalance in the dataset (also the fact that we could do both classification and regression and its unclear to me how to go about obtaining the best solution). I tried searching for "imbalanced" and found nothing that relates to this. Could you provide a link in case I have missed something that answers this precise question? – Nitesh Nov 12 '14 at 18:31
• heckman sample selection correction. The regression only applies to records where fraud is observed, controlling for the probability of fraud occuring. Same approach is (famously) taken to estimating labor supply equations when wages are only observed for workers. en.wikipedia.org/wiki/Heckman_correction – justin cress Nov 17 '14 at 16:36
• @justincress: Thanks for the comment. If you could go into more detail about how I could pose it as a supervised learning problem, it would be great. – Nitesh Nov 17 '14 at 21:02
• I don't really know how to recast econometrics into data science speak. It's just a two stage regression model where the first step is a probit and the second is OLS. To the extent that the models are specified directly (instead of learned from data) doesn't that make the algorithm "supervised" ? Not my expertise, sorry, but I think the heckman correction is exactly what you're looking for. – justin cress Nov 25 '14 at 14:23