# How to get probability of classification

I have binary classification, I tried several model KNN, SVM, decision tree and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works good, but I want to build some semi-supervised model because I have some unlabeled samples. In this case I need to get probability of classification what I tried, but it doesn't work.

As first I tried it for KNN

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))


I get [[1. 0.]]. I don't understand why it is 1? (as first I tought it is 100%, but I get it for all test samples)

Then I tried it for decision tree

classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))


I get [[1. 0.]] too. Why it is integer? Could someone help me? Thank you in advance.

• What if you set n_neighbors to be greater than 1 or max_depth in your DecisionTreeClassifier to a small integer? – oW_ Mar 20 '19 at 21:11
• @oW_ Great! Thank you that was the problem, KNN now works, but could you give me some hint, how to set the max_depth parameter? By default it is none, but I have no idea what number to put there. Thank you once more. – Max Mar 20 '19 at 21:18

The probability for KNN is the average of all the neighbors. If there is only one neighbor n_neighbor=1 it can only be 1 or 0.
The DecisionTreeClassifier expands until all the training data is classified perfectly if you don't control the depth. Again, this likely led to overfitting and to extreme probability predictions as a result. You should try different values for max_depth and see what works best. You can do say by performing cross validation. (If you are unfamiliar with these I recommend reading up on it first.)
• I tried a different values for max_depth, it is working now, but I still get some features which have 1. Can I think about this that they have 100% probability? – Max Mar 20 '19 at 22:45