0
$\begingroup$

I have a question about how many k values (k=1 or k=5 or k=50) to choose in the following two scenarios. I initially thought choosing k=5 will be the right choice of k for both because it will minimize the total number of errors. I would like to hear from experts what they think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios.

p1

p2

$\endgroup$
1

3 Answers 3

0
$\begingroup$

KNN is lazy learning at the beginning,Consider an extreme case, K=1, what will it happen? The training data will be perfectly predicted. The bias will be 0 when K=1, however, when it comes to new data (in test set), it has higher chance to be an error, which causes high variance. When we increase K, the training error will increase (increase bias), but the test error may decrease at the same time (decrease variance). We can think that when K becomes larger, since it has to consider more neighbors, its model is more complex.

We can see that when K is small, there are some outliers of 0 class are still 0 , and outliers of 1 class are still 1. When K becomes larger, the boundary is more consistent and reasonable.

The result on various Datasets shows that we could choose K around 13 or 20, which we’ll get the highest cross-validation accuracy

For a particular Dataset you can do

k_range = range(1,31)
k_scores = []
for k_number in k_range:
    knn = KNeighborsClassifier(n_neighbors=k_number)
    scores = cross_val_score(knn,X,y,cv=10,scoring='accuracy')
    k_scores.append(scores.mean())

plt.plot(k_range,k_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')

The above code will give you the graph of K vs Cross Validation Accuracy.You can select the value of k which gives you the highest validation accuracy for your dataset

$\endgroup$
2
  • $\begingroup$ Thanks! What do you think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios? $\endgroup$ Oct 10, 2021 at 14:16
  • $\begingroup$ K=1 will overfit the data .It will not perform well on test data . K=50 will not perform well on train dataset .K=5 is the best option among the three $\endgroup$ Oct 10, 2021 at 15:10
0
$\begingroup$

It depends on the goal of the project. Most machine learning projects want to maximize predictive ability. One useful way to maximize predictive ability is to pick an evaluation metric then find the value of k (a hyperparameter) that maximizes that evaluation metric on a hold-out data.

$\endgroup$
2
  • $\begingroup$ Hi Brian, thanks for your feedback! What do you think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios? $\endgroup$ Oct 10, 2021 at 14:43
  • $\begingroup$ Choosing k is an empirical question that should be answered through running the experiment I outlined in my answer. If I had to guess, I would pick k=5. $\endgroup$ Oct 10, 2021 at 14:57
0
$\begingroup$

For a KNN algorithm, it is wise not to choose k=1 as it will lead to overfitting. KNN is a lazy algorithm that predicts the class by calculating the nearest neighbor distance. If k=1, it will be that point itself and hence it will always give 100% score on the training data.

The best thing to do (and most of the people follow this) is to treat k as a hyperparameter and find it's value during the tuning phase as just by looking at the graph, one cannot determine what k value will be the best.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.