I’m trying to calculate probability of class 1. I’m using gradients boosting (catboost classifier) Is it normal to have an equal rate of positive classes in every predict_proba() bucket? e.g.:

[Probability] : Positive rate

[0 - 0.25] : 17%

[0.26 - 0.50] : 17%

[0.51 - 0.75] : 17%

[0.76 - 1] : 17%

However when I 'm using logistic regression and convert WOE to the scores, the distribution of class 1 rate decreases with increasing score. e.g.: [Score] : Positive rate

[100-150] : 17%

[151-200] : 15%

[200 - 250] : 13%

[250-300] : 10%

  • $\begingroup$ Might look unusual, but definitely possible that this happens. Could you provide the implementation you use of the algorithm and some information about the data you're training with? $\endgroup$ – Valentin Calomme Dec 4 '19 at 19:11
  • $\begingroup$ Having the same positive rate across all prediction buckets indicates the catboost model is not performing well. I don't understand exactly what you've done for the logreg reported distribution, could you elaborate? (It's odd that the positive rates there are all at most 17%, when the catboost results indicate that the global positive rate must be 17%; perhaps it indicates that the first bucket is much larger than the others?) $\endgroup$ – Ben Reiniger Dec 6 '19 at 14:54
  • $\begingroup$ This question cannot be answered properly without further context. $\endgroup$ – Peter Dec 6 '19 at 18:18

Based on the context provided this is what I assume you have done:

Cat boost: Predicted the probability on 1 using all your x variables and bucket the probabilities and calculate the percentage of 1s in each bucket.

Logistic Regression: You have used 1 of the x variables which was continuous and divided into the bands mentioned and calculated the incidence rate of 1 in each band.

If both the above statements are true then

Your catboost model most likely is either underfit or did not learn anything from the model. This inference is because the logistic Regression variables indicates that at least 1 variable has some prediction power on y.

It would be better to provide some additional details regarding the type of data and classes to analyze this further.

  • $\begingroup$ Solved. I noticed that indices were reset after converting y_test_true to DataFrame. And when I merged y_test_true and predict_proba X_test, I got random distribution of y_test_true. $\endgroup$ – mahomesII Dec 8 '19 at 7:54

Not the answer you're looking for? Browse other questions tagged or ask your own question.