I created a KNeighborsClassifier for my dataset adjusting the k hyper-parameter (the number of neighbors) in a for loop. The k value was between 1 and 20. The result was the graph below:

enter image description here

How do I interpret this graph? Which would be my best k value?


There's several ways that you can choose your k value for kNN -

You can use the common formula k = sqrt(n) where n is the number of data points in your training set or you can try choosing k where there is a good balance between computation expense vs noise.

Consider your what fits your problem: Do you care about runtime? The higher the k, the more expensive computationally it is to run.

It looks like around k= 6-10 you get some diminishing returns - you could set it there to get a good balance between noise and computation cost, but ultimately it is a very arbitrary selection, so pick what suits your use case best.

  • $\begingroup$ Isn't computation time mostly invariant of k? The distance metric still needs to be computed between all data points in the dataset. $\endgroup$ – Jayaram Iyer yesterday

The picture is telling you that test accuracy is improving with higher k, while train accuracy is degrading marginally. You can settle for k=15, because beyond that you have diminishing returns on the test accuracy.

In general, you want to select k that shoots for the highest test accuracy (unseen data).


That is called an Elbow-Curve! You need to look for the lowest train accuracy, or highest test accuracy, where the curve doesn't bend much more (y-axis), for a given increase in K (x-axis). In your case, it's around 7, but you could make an argument for 10, I suppose. As you increase your 'K' the accuracy improves, but only incrementally, and the cost (total compute resources, overall time to finish, etc.) will increase quite significantly.

Here is another example of a similar Elbow-Curve.

enter image description here

In this example, a good K would be around 7.

Here is some sample code for you to test, and further illustrate the point.

import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt
import numpy as np
#%matplotlib inline

df = pd.read_csv("C:\\your_path\\classified_data.csv",index_col=0)




from sklearn.preprocessing import StandardScaler
scaler = StandardScaler()

scaler.fit(df.drop('TARGET CLASS',axis=1))
scaled_features = scaler.transform(df.drop('TARGET CLASS',axis=1))

df_feat = pd.DataFrame(scaled_features,columns=df.columns[:-1])

from sklearn.model_selection import train_test_split
X = df_feat
y = df['TARGET CLASS']
X_train, X_test, y_train, y_test = train_test_split(scaled_features,df['TARGET CLASS'],
                                                    test_size=0.50, random_state=101)

from sklearn.neighbors import KNeighborsClassifier
knn = KNeighborsClassifier(n_neighbors=1)

KNeighborsClassifier(algorithm='auto', leaf_size=30, metric='minkowski',
           metric_params=None, n_jobs=1, n_neighbors=1, p=2,

pred = knn.predict(X_test)

from sklearn.metrics import classification_report,confusion_matrix


              precision    recall  f1-score   support

           0       0.88      0.90      0.89       250
           1       0.90      0.87      0.89       250

    accuracy                           0.89       500
   macro avg       0.89      0.89      0.89       500
weighted avg       0.89      0.89      0.89       500



print("Misclassification error rate:",round(np.mean(pred!=y_test),3))

error_rate = []
# Will take some time
for i in range(1,60):

    knn = KNeighborsClassifier(n_neighbors=i)
    pred_i = knn.predict(X_test)
    error_rate.append(np.mean(pred_i != y_test))

plt.plot(range(1,60),error_rate,color='blue', linestyle='dashed', marker='o',
         markerfacecolor='red', markersize=8)
plt.title('Error Rate vs. K Value', fontsize=20)
plt.ylabel('Error (misclassification) Rate',fontsize=15)

enter image description here

# sample data avalable here:
# https://www.kaggle.com/shubh247/simple-way-handle-classified-data-using-knn/data

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.