I have a question about how many k values (k=1 or k=5 or k=50) to choose in the following two scenarios. I initially thought choosing k=5 will be the right choice of k for both because it will minimize the total number of errors. I would like to hear from experts what they think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios.
$\begingroup$ Check this Value of k in k nearest neighbor algorithm $\endgroup$– seraliOct 10, 2021 at 7:48
$\begingroup$ This question shows little or no effort. A better question would be to at least investigate the scores produced from sklearn. Or, as @brian-spering states, choose an evaluation method and carry out nearest neighbor values. This way a debate could be formed on the 'correct' evaluation method. This question reminds me of the quote 'Without data, you're just another person with an opinion.' May I add without data your not doing science? $\endgroup$– mccurcioNov 10, 2022 at 13:34
It depends on the goal of the project. Most machine learning projects want to maximize predictive ability. One useful way to maximize predictive ability is to pick an evaluation metric then find the value of k (a hyperparameter) that maximizes that evaluation metric on a hold-out data set.
$\begingroup$ Hi Brian, thanks for your feedback! What do you think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios? $\endgroup$ Oct 10, 2021 at 14:43
$\begingroup$ Choosing k is an empirical question that should be answered through running the experiment I outlined in my answer. If I had to guess, I would pick k=5. $\endgroup$ Oct 10, 2021 at 14:57
KNN is lazy learning at the beginning,Consider an extreme case, K=1, what will it happen? The training data will be perfectly predicted. The bias will be 0 when K=1, however, when it comes to new data (in test set), it has higher chance to be an error, which causes high variance. When we increase K, the training error will increase (increase bias), but the test error may decrease at the same time (decrease variance). We can think that when K becomes larger, since it has to consider more neighbors, its model is more complex.
We can see that when K is small, there are some outliers of 0 class are still 0 , and outliers of 1 class are still 1. When K becomes larger, the boundary is more consistent and reasonable.
The result on various Datasets shows that we could choose K around 13 or 20, which we’ll get the highest cross-validation accuracy
For a particular Dataset you can do
k_range = range(1,31) k_scores =  for k_number in k_range: knn = KNeighborsClassifier(n_neighbors=k_number) scores = cross_val_score(knn,X,y,cv=10,scoring='accuracy') k_scores.append(scores.mean()) plt.plot(k_range,k_scores) plt.xlabel('Value of K for KNN') plt.ylabel('Cross-Validated Accuracy')
The above code will give you the graph of K vs Cross Validation Accuracy.You can select the value of k which gives you the highest validation accuracy for your dataset
$\begingroup$ Thanks! What do you think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios? $\endgroup$ Oct 10, 2021 at 14:16
$\begingroup$ K=1 will overfit the data .It will not perform well on test data . K=50 will not perform well on train dataset .K=5 is the best option among the three $\endgroup$ Oct 10, 2021 at 15:10
For a KNN algorithm, it is wise not to choose k=1 as it will lead to overfitting. KNN is a lazy algorithm that predicts the class by calculating the nearest neighbor distance. If k=1, it will be that point itself and hence it will always give 100% score on the training data.
The best thing to do (and most of the people follow this) is to treat k as a hyperparameter and find it's value during the tuning phase as just by looking at the graph, one cannot determine what k value will be the best.