Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods.

I'm trying to develop a classification model, where the desired outcome is the probability, rather than only the output class.

However, whatever variable/model combinations I try, the models are able to distinguish the bad cases quite nicely, but not the good cases. In other words, the distribution function of (smoothed) empirical outcomes vs the model estimated probabilities is not monotone, roughly having the shape of a tilted parabola (like letter $J$).

My question: What are the common strategies in order to reshape any model to focus the estimation on the good cases? If it is possible at all?

Intuitively, it seems that a possible strategy would be to define a custom metric for the minimizer, that would have asymmetric weights for good vs bad cases. I.e., in the case of penalized linear regression, the variable selection could then be skewed towards distinguishing goods instead of maximizing total AUC. But so far I'm unable to find any realized similar solutions. I guess, implementing something like that is not a trivial task..

Alternatively, is it possible to achieve this by transforming the input variables in a certain way?

  • $\begingroup$ What do you mean by being able to distinguish the bad cases nicely? If you can distinguish the bads, the not bads are goods. $\endgroup$ – David Masip Jun 12 '18 at 7:37
  • $\begingroup$ @DavidMasip this is in terms of estimated vs empirical probabilities, see i.e. the motivation behind Hosmer-Lemeshow test. Just here I'm talking about lack of monotonicity in the low-probability side, that is the "good" cases. Or in other words - low sensitivity under certain cut-offs. $\endgroup$ – Nutle Jun 12 '18 at 8:17
  • $\begingroup$ I'll add that it is fairly clearly described in this article by Esarey and Pierce, who have also implemented their proposed tests with heatmapFit package. $\endgroup$ – Nutle Jun 12 '18 at 8:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.