Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
I have a question about how many k values (k=1 or k=5 or k=50) to choose in the following two scenarios. I initially thought choosing k=5 will be the right choice of k for both because it will minimize the total number of errors. I would like to hear from experts what they think is the best k to choose out of the three options k=1 or k=5 or k=50 for these particular two scenarios.
It depends on the goal of the project. Most machine learning projects want to maximize predictive ability. One useful way to maximize predictive ability is to pick an evaluation metric then find the value of k (a hyperparameter) that maximizes that evaluation metric on a hold-out data set.
KNN is lazy learning at the beginning,Consider an extreme case, K=1, what will it happen? The training data will be perfectly predicted. The bias will be 0 when K=1, however, when it comes to new data (in test set), it has higher chance to be an error, which causes high variance. When we increase K, the training error will increase (increase bias), but the test error may decrease at the same time (decrease variance). We can think that when K becomes larger, since it has to consider more neighbors, its model is more complex.
We can see that when K is small, there are some outliers of 0 class are still 0 , and outliers of 1 class are still 1. When K becomes larger, the boundary is more consistent and reasonable.
The result on various Datasets shows that we could choose K around 13 or 20, which we’ll get the highest cross-validation accuracy
For a particular Dataset you can do
k_range = range(1,31)
k_scores = []
for k_number in k_range:
knn = KNeighborsClassifier(n_neighbors=k_number)
scores = cross_val_score(knn,X,y,cv=10,scoring='accuracy')
k_scores.append(scores.mean())
plt.plot(k_range,k_scores)
plt.xlabel('Value of K for KNN')
plt.ylabel('Cross-Validated Accuracy')
The above code will give you the graph of K vs Cross Validation Accuracy.You can select the value of k which gives you the highest validation accuracy for your dataset
For a KNN algorithm, it is wise not to choose k=1 as it will lead to overfitting. KNN is a lazy algorithm that predicts the class by calculating the nearest neighbor distance. If k=1, it will be that point itself and hence it will always give 100% score on the training data.
The best thing to do (and most of the people follow this) is to treat k as a hyperparameter and find it's value during the tuning phase as just by looking at the graph, one cannot determine what k value will be the best.
