1
$\begingroup$

How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted?

Here x axis shows probabilites of target being 1 and y axis are the number of instances with that probability.

enter image description here

Probabilities for the target of the train set is as follows:

enter image description here

$\endgroup$
4
  • $\begingroup$ So these are the predicted probabilities for a binary variable for a test set? $\endgroup$ Feb 23, 2022 at 10:51
  • $\begingroup$ @user2974951 yes, exactly $\endgroup$
    – gato
    Feb 23, 2022 at 10:54
  • $\begingroup$ Then no, you cannot tell much just from this data alone. You would need the train data probabilities as well, to compare. $\endgroup$ Feb 23, 2022 at 10:56
  • $\begingroup$ @user2974951 added the plot for train probabilities too $\endgroup$
    – gato
    Feb 23, 2022 at 11:02

1 Answer 1

2
$\begingroup$

It is expected to have "more confident" predicted probabilities in the train set, i.e your model will assign probabilities closer to 0 and 1 and less in the middle for the samples used to train.

As mentioned above, you hardly state something about your model performance just by probabilities' histograms, but you could add more information by splitting them by positive and negative class and adding a metric of separateness like ks

Code example:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import ks_2samp

import matplotlib.pyplot as plt

data = load_breast_cancer()
X = data.data
y = data.target
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = .2, random_state = 42)

clf = RandomForestClassifier(random_state = 42).fit(X_train,y_train)


preds_test = clf.predict_proba(X_test)[:,1]
preds_train = clf.predict_proba(X_train)[:,1]

ks_train = round(ks_2samp(preds_train[y_train == 1], preds_train[y_train == 0])[0],4)
ks_test = round(ks_2samp(preds_test[y_test == 1], preds_test[y_test == 0])[0],4)


fig, ax = plt.subplots(1,2,figsize = (10,5))

ax[0].hist(preds_train[y_train == 1], color = "darkred",bins = "scott", alpha = .5, edgecolor = "red")
ax[0].hist(preds_train[y_train == 0], color = "darkgreen",bins = "scott", alpha = .5, edgecolor = "green")
ax[0].set_title(f"Class separation in train set\nks: {ks_train}")

ax[1].hist(preds_test[y_test == 1], color = "darkred",bins = "scott", alpha = .5, edgecolor = "red")
ax[1].hist(preds_test[y_test == 0], color = "darkgreen",bins = "scott", alpha = .5, edgecolor = "green")
ax[1].set_title(f"Class separation in test set\nks: {ks_test}");

enter image description here

$\endgroup$
1
  • $\begingroup$ you are saying the information I provide is not enough to tell whether the model is overfitted or not. I thought since train probabilities were so certain on the edges, it should be overfitted. In your answer how do you interpret the plots of the train set and test set? What does class separation in them tell us? $\endgroup$
    – gato
    Feb 24, 2022 at 7:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.