0
$\begingroup$

I have binary classification, I tried several model KNN, SVM, decision tree and random forest. I have 50 000 samples, X_train has 50 000 rows and 2300 columns. Everything works good, but I want to build some semi-supervised model because I have some unlabeled samples. In this case I need to get probability of classification what I tried, but it doesn't work.

As first I tried it for KNN

from sklearn.neighbors import KNeighborsClassifier
classifier = KNeighborsClassifier(n_neighbors = 1, metric = 'minkowski', p = 2)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]]. I don't understand why it is 1? (as first I tought it is 100%, but I get it for all test samples)

Then I tried it for decision tree

classifier = DecisionTreeClassifier(random_state=0)
classifier.fit(X_train, y_train)
y_pred = classifier.predict(X_test)
print(classifier.predict_proba(X_test[0]))

I get [[1. 0.]] too. Why it is integer? Could someone help me? Thank you in advance.

$\endgroup$
  • $\begingroup$ What if you set n_neighbors to be greater than 1 or max_depth in your DecisionTreeClassifier to a small integer? $\endgroup$ – oW_ Mar 20 '19 at 21:11
  • $\begingroup$ @oW_ Great! Thank you that was the problem, KNN now works, but could you give me some hint, how to set the max_depth parameter? By default it is none, but I have no idea what number to put there. Thank you once more. $\endgroup$ – Max Mar 20 '19 at 21:18
0
$\begingroup$

It is indeed a probability of 1 because you didn't change the default parameters.

The probability for KNN is the average of all the neighbors. If there is only one neighbor n_neighbor=1 it can only be 1 or 0.

The DecisionTreeClassifier expands until all the training data is classified perfectly if you don't control the depth. Again, this likely led to overfitting and to extreme probability predictions as a result. You should try different values for max_depth and see what works best. You can do say by performing cross validation. (If you are unfamiliar with these I recommend reading up on it first.)

| improve this answer | |
$\endgroup$
  • $\begingroup$ I tried a different values for max_depth, it is working now, but I still get some features which have 1. Can I think about this that they have 100% probability? $\endgroup$ – Max Mar 20 '19 at 22:45
  • $\begingroup$ what do you mean "features which have 1"? predictions? maybe one of your input features is perfectly correlated with your target variable. you can interpret this as your model having 100% confidence in its prediction. If you want "real" probabilities you should look into "probability calibration". scikit-learn has tools for it. just search for it. $\endgroup$ – oW_ Mar 20 '19 at 23:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.