# Probability Distribution on the Test Set

How can one interpret the probability distibution of the predictions for the target of the test set? For example if we wanted to interpret the plot below, can we say it is overfitted?

Here x axis shows probabilites of target being 1 and y axis are the number of instances with that probability.

Probabilities for the target of the train set is as follows:

• So these are the predicted probabilities for a binary variable for a test set? Feb 23, 2022 at 10:51
• @user2974951 yes, exactly
– gato
Feb 23, 2022 at 10:54
• Then no, you cannot tell much just from this data alone. You would need the train data probabilities as well, to compare. Feb 23, 2022 at 10:56
• @user2974951 added the plot for train probabilities too
– gato
Feb 23, 2022 at 11:02

It is expected to have "more confident" predicted probabilities in the train set, i.e your model will assign probabilities closer to 0 and 1 and less in the middle for the samples used to train.

As mentioned above, you hardly state something about your model performance just by probabilities' histograms, but you could add more information by splitting them by positive and negative class and adding a metric of separateness like ks

Code example:

from sklearn.datasets import load_breast_cancer
from sklearn.model_selection import train_test_split
from sklearn.ensemble import RandomForestClassifier
from scipy.stats import ks_2samp

import matplotlib.pyplot as plt

X = data.data
y = data.target
X_train, X_test, y_train, y_test = train_test_split(X,y,test_size = .2, random_state = 42)

clf = RandomForestClassifier(random_state = 42).fit(X_train,y_train)

preds_test = clf.predict_proba(X_test)[:,1]
preds_train = clf.predict_proba(X_train)[:,1]

ks_train = round(ks_2samp(preds_train[y_train == 1], preds_train[y_train == 0])[0],4)
ks_test = round(ks_2samp(preds_test[y_test == 1], preds_test[y_test == 0])[0],4)

fig, ax = plt.subplots(1,2,figsize = (10,5))

ax[0].hist(preds_train[y_train == 1], color = "darkred",bins = "scott", alpha = .5, edgecolor = "red")
ax[0].hist(preds_train[y_train == 0], color = "darkgreen",bins = "scott", alpha = .5, edgecolor = "green")
ax[0].set_title(f"Class separation in train set\nks: {ks_train}")

ax[1].hist(preds_test[y_test == 1], color = "darkred",bins = "scott", alpha = .5, edgecolor = "red")
ax[1].hist(preds_test[y_test == 0], color = "darkgreen",bins = "scott", alpha = .5, edgecolor = "green")
ax[1].set_title(f"Class separation in test set\nks: {ks_test}");


• you are saying the information I provide is not enough to tell whether the model is overfitted or not. I thought since train probabilities were so certain on the edges, it should be overfitted. In your answer how do you interpret the plots of the train set and test set? What does class separation in them tell us?
– gato
Feb 24, 2022 at 7:00