2
$\begingroup$

I am currently working on a credit risk related project where i built a binary logistic regression model for an imbalanced dataset.

According to the regulations i have to prove that the model performs well on different subsets of data (e.g. age-group [18, 25] compared to age-group [26, 40], mortgages compared to consumer loans, high/low income). Typically, the sub-Segments would be indicated by a binary variable, but it could also be that there are more than two Segments to compare.

I spent the whole day looking for possible solutions but so far I did not find something particular useful for this challenge. Unfortunately, it is not enough to show that AUC does not drop significantly on each of the sub-segments.

Are there any of you who already have experience with these kind of problems?

Thank you very much!

$\endgroup$
5
  • $\begingroup$ The question is what does the regulation call "the model performs well"? I would imagine that it's something like evaluating on these subsets of data and proving that there is no significant difference in performance between subsets or between a subset and the general case. In this case significance tests might be relevant. $\endgroup$ – Erwan Nov 3 '20 at 0:15
  • $\begingroup$ performing well in this sense means, that customers are ranked properly according to their probability of default. $\endgroup$ – matosch Nov 6 '20 at 8:53
  • $\begingroup$ ok but that's not a formal criterion, is there a precise threshold for "ranked properly"? Maybe something like Spearman correlation against gold standard higher than X? I assume that nobody expects a perfect model. $\endgroup$ – Erwan Nov 6 '20 at 11:38
  • $\begingroup$ Could you elaborate on why "it is not enough to show that AUC does not drop significantly on each of the sub-segments."? $\endgroup$ – Ben Reiniger Nov 6 '20 at 15:12
  • $\begingroup$ In what country are you ? can you provide the local reglementation ? $\endgroup$ – lcrmorin Mar 31 at 13:55