3
$\begingroup$

The KNeighborsClassifier has a method for predicting class probabilities. However, I cannot find any documentation describing how these probabilities are computed.

Here is a toy example that returns class probabilites:

from sklearn.neighbors import KNeighborsClassifier
import numpy as np
N = 100
np.random.seed(1)
X = np.random.random((N,2))
Y = np.random.randint(2, size=N)
model = KNeighborsClassifier(5, weights='distance')
model.fit(X, Y)
print(model.predict_proba([[0.5, 0.5]]))
#>>> [[0.55486525 0.44513475]]
$\endgroup$
4
$\begingroup$

The class probabilities are the normalized weighted average of indicators for the k-nearest classes, weighted by the inverse distance.

For example: Say we have 6 classes, and the 5 nearest examples to our test input have class labels 'F', 'B', 'D', 'A', and 'B', with distances 2, 3, 4, 5, and 6, respectively.

Then the unnormalized class probabilities can by computed by:

(1/2) * [0, 0, 0, 0, 0, 1] + (1/3) * [0, 1, 0, 0, 0, 0] + 
(1/4) * [0, 0, 0, 1, 0, 0] + (1/5) * [1, 0, 0, 0, 0, 0] + 
(1/6) * [0, 1, 0, 0, 0, 0] =

[1/5 ,1/2, 0, 1/4, 0, 1/2]

And after normalizing we get:

[0.13793103, 0.34482758, 0.0, 0.17241379, 0.0, 0.34482758]

If you choose weights='uniform' then it becomes a simple average, which can be computed by multiplying each indicator by (1/k) and summing, with no need for additional normalization.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.