1
$\begingroup$

Running sklearn.KNeighborsClassifier() on Kaggle's Leaf Classification sample (set of 99 species, 10 specimen each), with defaults kNN parameters and a grid search optimization (using sklearn.model_selection.GridSearchCV), I find the best accuracy is achieved for k=1.

Although this is a technically valid choice of k, I feel using a single neighbor means the model wasn't able to find a strong relationship between neighbors.

Am I right, or is k=1 sometimes a good choice, and for which kind of problems ?

Note that using neg_log_loss as scoring parameter leads to an optimal number of k=5 neighbors. I am still struggling to understand how this is an improvement since the accuracy is smaller then.

Update : I was able to find more information using the "1NN" and "1-NN" keywords, which suggest this is indeed a good choice in some situations.

$\endgroup$
0
2
$\begingroup$

Having k = 1 is not inherently unreasonable. This just means that all new observations will be predicted to be the class label of the first closest neighbour you have in your training set. Predicted class probabilities should just be 1 or 0 since the the estimator will take on the form 1/1 or 0/1. This also makes it clearer as to why log loss will probably not favor a k = 1 model; the predicted probabilities will be far too extreme in the cases where the model favors the wrong class label. Consider the case where the the true label is 0, i.e. $y_i = 0$. If the closest neighbour in your training set is of class label 1, then your model will incorrectly predict $Pr[Y = 1|X] = 1.$ Hence, $ (1-y_i) * ln(1-Pr[Y = 1|X])$ will be infinity. On the other hand, if the true label is 1, i.e., $y_i = 1$, and the closest neighbour to this observation ends up being of label 0, now $Pr[Y = 1|X] = 0$ and we have $y_i * ln(Pr[Y = 1|X])$ infinity as well. Thus, log loss will either be 0 (where your model predicts correctly) or -infinity for all of your observations; clearly very extreme.

However, I would just like to discuss the validity of even using accuracy as a scoring measure in the first place. Accuracy is simply not a proper scoring rule and often leads to unoptimal, bogus models. The problem is due to the arbitrary nature of the threshold you use to classify a "1" or a "0", and also that a correct classification of "1" with predicted probability 51% is not the same as a correct classification of "1" with predicted probability 90% (for instance).

Logloss is a proper scoring rule that is minimized when the predicted probabilities are close to the true, population probabilities. Accuracy is clearly influenced by what threshold you choose to use and I would not be surprised at all to see the accuracy seemingly "higher" if you choose the right threshold that maximizes accuracy for the model fit on log loss (assuming you are just using the very arbitrary 50% cutoff). Thresholds chosen should depend on the context of how your model is being used and perhaps one's own personal belief in risk; in particular, the costs of making incorrect predictions.

If you are absolutely forced in classifying objects as 1 or 0, then I suggest tuning the threshold as part of your model validation process and then assessing the accuracy on a held out test set (using the threshold found in your validation). See this thread here for more discussion on this topic.

$\endgroup$
1
  • $\begingroup$ Thanks for this answer. Although it doesn't really address the main question, it clearly answers the secondary one. As for accuracy as a scoring measure, actually I had already read a handful of articles (including Frank Harrell's) but your formulation provides an insightful perspective. I'm not sure how the threshold part applies to multiclass classification, though ? $\endgroup$ Apr 21 '19 at 9:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.