Remedie for a stubborn recall result?

Question

I was working on a project connected to predicting default on credit loan with 0-1 loss. The recall is a crucial measure that should be maximized in this case, while monitoring precision for sanity of the results. As it is usuall with such data, the sample was heavily unbalanced, with low frequency of default cases. Although the algorithms I have trained (bagged/boosted trees, logistic regression etc.) obtained high accuracy and high precision the recall was low.

I have tried to work with probability treshold of labeling predicted cases, namely decreasing it, but a tiny increase in recall was highly costly in terms of precision and accuracy.

Am I doomed because those default cases that were not recalled are inseparable, or is there a trick to work with that?
I have thought of maybe doing some clustering on the data and fitting different sub models for different clusters, but I am not sure how to proceed.
I have also thought of subsampling in such a way that the default and not default would be more less balanced, and repeating such procedure keeping the default cases fixed and not default changing over the sample. Later creating an ensamble.

Answer 1

Maybe I'm not aware of it, but I seriously doubt that any such trick exists ;)

It's not surprising that any small increase in recall causes a massive drop in precision in this scenario. As an analogy, you're buying something rare and expensive with something common and cheap, so you must give away a lot of the latter just to acquire a little of the former. I think that all the methods based on resampling will more or less cause the same problem, even though some improve things slightly.

It's not at all guaranteed to work better, but I'd suggest trying methods like anomaly detection or one class classification.

Imho it might also be interesting to transform this into a regression problem, where the goal is not to really predict the event of default but quantify the risk of default happening. It won't perform better of course, but I think that this setting is more realistic: what happens is that many people are at risk of default, but only some of them will actually default by chance or at least for reasons which are unlikely to be in the data.