We can get the prediction probabilities of a binary classifier from sklearn's API using the predict_proba method. Is it reasonable to expect that the shape of a histogram plotted for the prediction probabilities of let's say the '1' class to approximate a normal distribution? What is the statistical theory that allows for this? I noticed this instance for a logistic regression model I trained.
1
-
$\begingroup$ I think it is better if you ask this question on stats.stackexchange.com. Datascience exchange is for DS related things, while stats is for statistics related. Your question seems more to be statistics related. $\endgroup$– Ankit SethJan 27 at 9:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
The landscape of the output probabilities depends entirely on the training data. If the data itself is sampled from a normal distribution, then the learned probabilities will reflect that. Otherwise, no.
Therefore, in the general case, no, it is not reasonable to expect the shape of a histogram plotted from the prediction probabilities to approximate a normal distribution.
-
$\begingroup$ Thanks for you feedback @noe, and your observation makes sense. That said, I did find a similar case under stack exchange-crossvalidated: stats.stackexchange.com/questions/576151/…. It suggests that the better the classifier performs the more skewed one should expect the distribution to be. Do you agree? I know you noted that if the data was sampled from a normal distribution, the probabilities should reflect that. Does this assertion go against what the link suggests? $\endgroup$– zebinxJan 27 at 20:50
-
$\begingroup$ I think that in the linked question they used the term "normal" as in "usual", referring to the close match between predictions and true values; I don't think they refer to the Gaussian distribution. $\endgroup$– noeJan 27 at 21:15
-
$\begingroup$ Also, the skewness of the distribution of predicted probabilities can be very high (the model is "very sure") and yet the accuracy be very low. $\endgroup$– noeJan 27 at 21:17
-
$\begingroup$ Probably worth mentioning the beta distribution (en.wikipedia.org/wiki/Beta_distribution), as a specific distribution over probabilities that naturally describes both the certain and uncertain regimes for classifiers. $\endgroup$– bogovicjJan 29 at 17:30