3
$\begingroup$

Assume a binary classification problem, with $1$ denoted as a "bad" outcome, and $0$ as a "good" outcome. If it's relevant, in the sample there are significantly more bads than goods.

I'm trying to develop a classification model, where the desired outcome is the probability, rather than only the output class.

However, whatever variable/model combinations I try, the models are able to distinguish the bad cases quite nicely, but not the good cases. In other words, the distribution function of (smoothed) empirical outcomes vs the model estimated probabilities is not monotone, roughly having the shape of a tilted parabola (like letter $J$).

My question: What are the common strategies in order to reshape any model to focus the estimation on the good cases? If it is possible at all?

Intuitively, it seems that a possible strategy would be to define a custom metric for the minimizer, that would have asymmetric weights for good vs bad cases. I.e., in the case of penalized linear regression, the variable selection could then be skewed towards distinguishing goods instead of maximizing total AUC. But so far I'm unable to find any realized similar solutions. I guess, implementing something like that is not a trivial task..

Alternatively, is it possible to achieve this by transforming the input variables in a certain way?

Improve this question
$\endgroup$
3
  • $\begingroup$ What do you mean by being able to distinguish the bad cases nicely? If you can distinguish the bads, the not bads are goods. $\endgroup$
    – David Masip
    Jun 12, 2018 at 7:37
  • 1
    $\begingroup$ @DavidMasip this is in terms of estimated vs empirical probabilities, see i.e. the motivation behind Hosmer-Lemeshow test. Just here I'm talking about lack of monotonicity in the low-probability side, that is the "good" cases. Or in other words - low sensitivity under certain cut-offs. $\endgroup$
    – runr
    Jun 12, 2018 at 8:17
  • 1
    $\begingroup$ I'll add that it is fairly clearly described in this article by Esarey and Pierce, who have also implemented their proposed tests with heatmapFit package. $\endgroup$
    – runr
    Jun 12, 2018 at 8:29

1 Answer 1

Reset to default
2
$\begingroup$

I think what you are dealing with is a type I vs. type II error discussion. It seems that you want to avoid Type II error.

One way to go is to consider $F_\beta$ for your performance metric. It is a modified version of $F_1$. As seen here : https://en.wikipedia.org/wiki/F1_score, $F_\beta$ can be formulated in terms of type I /type II error. You then need to select a $\beta$ based on how much avoiding type II error is important to you. This can be done by considerations on the cost of type II errors compared to the cost of type I error.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.