7
$\begingroup$

I have a dataset with around 900.000 records, around 1000 of which are marked as positive (the studied event occurred).

The probability of the event occurring is always low (i.e. < 0.1), and I would like to create a regression model to predict the probability of the event occurring.

My first thought was to use logistic regression, but I am not sure if I could directly interpret the output as the probability of the event happening. The same doubt arises when using other models, such as SVM or RF.

Another doubt would be whether usual evaluation metrics (e.g. RMSE) would work well on such a model, since even a predictor that always outputs 0 would have a very good score.

Improve this question
$\endgroup$
1

2 Answers 2

Reset to default
5
$\begingroup$

The output of logistic regression is exactly that - the probability of an event happening. If your covariates are informative then your model will do better than just saying "P=1000/900000" everytime, because it might say "P=10000/900000" for a positive event, or even "P=0.9" of a positive event given certain covariates.

If there's no predictive power in the covariates (ie they don't correlate with the positive events) then yes, the model will say P=1000/900000 but that's the way it is.

If you want a binary outcome then you have to decide what threshold of P you choose, based on your fear of false positives/negatives or value of true positives/true negatives. For example, if one false positive means people die, you set your threshold such that you don't get false positives. These thresholds are generally not statistical decisions but based on application logic. ROC curves can help here.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your help. The point is: I'm just predicting the probabilities, not doing any actual classification. So the biggest problem is how to evaluate the model, since the common methods expect classification outputs and not probabilities. As far as I know ROC curves work well only when the outcomes are binary, which is not my case. $\endgroup$
    – MDG
    Mar 7, 2017 at 14:31
  • $\begingroup$ You said your outcomes are "positive (the studied event occurred)" which sounds like binary to me. ROC curves show how specificity and sensitivity vary as you change the threshold, so you can evaluate your model at any threshold against how bad you feel about type-I and type-II errors. $\endgroup$
    – Spacedman
    Mar 7, 2017 at 14:35
  • $\begingroup$ I think the problem description may be a little confusing, I'm sorry for that. What I meant is that yes, I know the binary outcome in the training and testing datasets. However, the probabilities of the event happening are always very low (i.e. 1/100 to 1/100000), so I am not trying to predict whether it would happen in a timeframe, but I am only trying to predict how likely it would be to happen. For this reason, I am not sure if putting a threshold would be a good idea, as the eventual positive outputs would be misleading. Or am I wrong? $\endgroup$
    – MDG
    Mar 7, 2017 at 14:47
  • $\begingroup$ Timeframe? Where did that come in. Is it not like this: you have 900,000 people, and 1000 of them have an illness. A null model predicts P(illness) as 1000/900,000. A logistic regression might notice that all those 1000 illnesses are in people over 60, and all the 899000 are in people under 60, and so will predict P(illness) as 1 for any person over 60, and P(illness)=0 for anyone under. In reality you'll get a varying probability that might only get up to P(illness)=0.1. Well, no amount of desire for strong results will get them if that information is not in the data. No free lunches. $\endgroup$
    – Spacedman
    Mar 7, 2017 at 15:24
  • $\begingroup$ Thank you for your answer. Yes, you are right. In your analogy, if the probability of illness gets only up to P(illness)=0.1, then I think it would be useless to set a threshold to output positive values. For this reason, my aim would be to just try to give the most accurate probability estimate, e.g. just saying that P(illness) = 0.001 for a specific person. The only thing is that I don't know how I should evaluate such a model, since of course usual binary evaluation metrics don't work (since the classification should always be 0 anyway). My only idea is the logistic loss, but I'm not sure $\endgroup$
    – MDG
    Mar 7, 2017 at 15:48
0
$\begingroup$

This type of dataset is commonly called a skewed dataset. This is when a certain class is over-represented in contrast to the other labels in the dataset.

There are many algorithms that are well suited to these types of problems. Specifically, anomaly detection algorithms are capable of learning the distribution a single set of labels (event not occurring) and then it will be able to flag when an anomaly occurs (event occurs). This is when an instance is sufficiently beyond the learned distribution. You can use this to get the probability of an event occurring based on a p-statistic test using the feature-space you have set up in contrast with those from your learned distribution.

The simplest method would be doing a generalized likelihood ratio test (GLRT). But, I think you will most likely find more luck using a K-NN based method for skewed datasets.

Since you have a lot of data you can use ranking algorithms to generate your p-statistics. Look into rankSVM.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.