1
$\begingroup$

I have the binary classification, I tried several models KNN, SVM, decision tree, and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works well, but I want to build some semi-supervised model because I have some unlabeled samples. In this case, I need to get the probability of classification that I tried, but it doesn't work.

At first, I tried it for KNN

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]]. I don't understand why it is 1? (as first I thought it is 100%, but I get it for all test samples)

Then I tried it for the decision tree

classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]] too. Why it is an integer?

$\endgroup$
2
  • $\begingroup$ What if you set n_neighbors to be greater than 1 or max_depth in your DecisionTreeClassifier to a small integer? $\endgroup$
    – oW_
    Mar 20, 2019 at 21:11
  • $\begingroup$ @oW_ Great! Thank you that was the problem, KNN now works, but could you give me some hint, how to set the max_depth parameter? By default it is none, but I have no idea what number to put there. Thank you once more. $\endgroup$
    – Max
    Mar 20, 2019 at 21:18

1 Answer 1

0
$\begingroup$

It is indeed a probability of 1 because you didn't change the default parameters.

The probability for KNN is the average of all the neighbors. If there is only one neighbor n_neighbor=1 it can only be 1 or 0.

The DecisionTreeClassifier expands until all the training data is classified perfectly if you don't control the depth. Again, this likely led to overfitting and to extreme probability predictions as a result. You should try different values for max_depth and see what works best. You can do say by performing cross validation. (If you are unfamiliar with these I recommend reading up on it first.)

$\endgroup$
2
  • $\begingroup$ I tried a different values for max_depth, it is working now, but I still get some features which have 1. Can I think about this that they have 100% probability? $\endgroup$
    – Max
    Mar 20, 2019 at 22:45
  • $\begingroup$ what do you mean "features which have 1"? predictions? maybe one of your input features is perfectly correlated with your target variable. you can interpret this as your model having 100% confidence in its prediction. If you want "real" probabilities you should look into "probability calibration". scikit-learn has tools for it. just search for it. $\endgroup$
    – oW_
    Mar 20, 2019 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.