I am dealing with an unbalanced binary classification problem. The problem is so unbalanced (2:98) and hard to predict that I am interested in probability of the positive outcome instead of trying to predict the actual binary output. Depending on the model used this require either to calibrate the model in probabilities (tree based models) or transforming scores into probabilities using some spline (NN).

But in the end, for all practical matters I use buckets of probabilities. With 2% being the average I use predefined buckets like these : [0,0.5%],[0.5%,1%],[1%,2%],[2%,5%],[5%,10%],[10%,25%],[25%,50%],[50%,100%]. To attribute a bucket to an instance, the simple way to go is to just put the instance in the bucket that match the predicted probability. I am aware this might not be optimal (removing information on the output, not using the optimal buckets ... etc.) but let's say those buckets are an expert-based constraint (or enforced by law in some cases).

I have some doubts on the necessity to go trough individual probability to predict a bucket. I also have the feeling that directly predicting buckets instead of probabilities could help remove a bit of the model instability I am observing (basically different random initialisation for NN might yield very different individual output - I am usually painfully dealing with this trough regularisation).

To get a general solution I was trying to find some loss function that would match this ordinal/multi-class/calibrated in probabilities problem, but couldn't find anything. So I am asking you: is there a general approach to predict given buckets of probabilities ? (Feel free to give model specific approaches instead / or give an explanation on why this might be a bad idea.)

  • 2
    $\begingroup$ Not sure about the model selection, but I doubt that using buckets increases the stability of the model. Mapping the continuous variable into a finite set of values makes the model LESS robust and stable -- think about what happens when the sample is near the edge. Without mapping to a discrete set, the model is way more stable under perturbations. $\endgroup$ Dec 7 '20 at 6:15
  • $\begingroup$ Mmmh i am not sure that intuition about edge cases would generalise well to everything... say I have two model originating from different random initialisation: one is outputting 12% for a given instance and the other 18%. Mapping directly to bucket would give [10%,25%] twice... not sure this would be better in general (loosing information about the output). But it does seems to match some stability (or apparent stability) criteria. $\endgroup$
    – lcrmorin
    Dec 7 '20 at 9:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.