# How to get probability of classification

I have the binary classification, I tried several models KNN, SVM, decision tree, and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works well, but I want to build some semi-supervised model because I have some unlabeled samples. In this case, I need to get the probability of classification that I tried, but it doesn't work.

At first, I tried it for KNN

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test))


I get [[1. 0.]]. I don't understand why it is 1? (as first I thought it is 100%, but I get it for all test samples)

Then I tried it for the decision tree

classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test))


I get [[1. 0.]] too. Why it is an integer?

• What if you set n_neighbors to be greater than 1 or max_depth in your DecisionTreeClassifier to a small integer?
– oW_
Mar 20, 2019 at 21:11
• @oW_ Great! Thank you that was the problem, KNN now works, but could you give me some hint, how to set the max_depth parameter? By default it is none, but I have no idea what number to put there. Thank you once more.
– Max
Mar 20, 2019 at 21:18

The probability for KNN is the average of all the neighbors. If there is only one neighbor n_neighbor=1 it can only be 1 or 0.
The DecisionTreeClassifier expands until all the training data is classified perfectly if you don't control the depth. Again, this likely led to overfitting and to extreme probability predictions as a result. You should try different values for max_depth and see what works best. You can do say by performing cross validation. (If you are unfamiliar with these I recommend reading up on it first.)
• I tried a different values for max_depth, it is working now, but I still get some features which have 1. Can I think about this that they have 100% probability?