0
$\begingroup$

I have some question related to quasi complete seperation problem on logistic regression algorithm.

So i run the model to predict credit risk and turns out it gave me good prediction score (AUC around 95%) and the gap between auc train and test not wide. But turns out it have quasi complete seperation problem. So is that a problem and make my model bias/unstable?

I also trying to make other model and handled the quasi complete seperation problem by removing some feature and it gave me AUC score around 76% but no more complete quasi seperation problem. So which model should i choose?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

See How to deal with perfect separation in logistic regression?

Perfect separation tells you that there is a combination of predictor variables that perfectly predicts if someone is going to default on their loans. For example, an income of less than \$X, a monthly expenditure of more than \$Y, and owning Z might perfectly predict that someone will default, and vice versa. This generally implies that there's something wrong with your classification model, because well-off people default on their loans all the time because of human error, and financially insecure people can still pay their interest payments on time.

Here are some suggestions from my experience:

  1. How are you defining credit risk?

This sounds obvious, but is your definition based on a quantitative score you're calculating from your predictor variables? If so, try adjusting the weights and/or the threshold. Also, are you predicting credit risk for individuals or firms? Using predictor variables that are known to be correlated well with credit risk and validated by academic research can help you to build a more empirically reliable model.

  1. Do you have multicollinear variables?

Multicollinearity refers to predictor variables that are strongly correlated with one another. I've seen pairs of variables with an R^2 of more than 0.99. This tends to distort your coefficients and output because they effectively measure the same construct. I don't think there's much of a consensus on which variable to remove because it's a qualitative decision, so you could select the more interpretable variable or the variable that's better supported by past models.

  1. Did you standardize your variables?

Use a standard scaler on your numerical variables. I've worked with models that took a company's stock market capitalization (which can go up to a few billion dollars) and the ratio between the company's net profit and total assets on hand (which tends to range from 0 to 5). In cases like these, standardization ensures that the distribution of your predictor variables remains the same while ensuring that the range of possible values are relatively constant between variables.

Improve this answer
$\endgroup$
1
  • $\begingroup$ thank you for ur answer. Actually i've already successfully handle the perfect seperation problem but my question is which model should i choose? The one that have AUC score 95% (without handling perfect seperation) or the model after handling perfect seperation that have AUC score only around 76%? Because perfect seperation make the model perform better score but cannot be interpreted through the feature parameter. $\endgroup$
    – Jovian Aditya
    Dec 15, 2022 at 0:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.