# Massive difference in accuracy of KNN depending on random_state

pardon the noob question but I am baffled by the following behavior. My model has MASSIVELY different results based on the random seed. I want to train a KNN classifier on the famous Kaggle Titanic problem where we attempt to predict survival or not. I focus only on the "Sex" feature, to make things easier.

The problem becomes now that by changing the random seed the results of the accuracy change incredibly. For example, one random seed gives me a score of 0.78 and another random seed gives me a score of 0.17, and different random seeds give more or less everything in between. How can this significant change in score behavior be explained? Also, why does this change in accuracy become less significant when the n_neighbors = 2 or above? Thanks in advance! Here is the code in question.

df_sex = df[["Sex"]]
y = df["Survived"]

def training(state):
X_train, X_test, y_train, y_test = train_test_split(df_sex, y, random_state=state )
knn = KNeighborsClassifier(n_neighbors=1)
knn.fit(X_train, y_train)

print("Test set score: {:.2f}".format(knn.score(X_test, y_test)))
print("Train set score: {:.2f}".format(knn.score(X_train, y_train)))

training(0)
training(3)
training(10)
training(21)


gives output

Test set score: 0.78
Train set score: 0.79
Test set score: 0.22
Train set score: 0.21
Test set score: 0.17
Train set score: 0.23
Test set score: 0.77
Train set score: 0.79

• You are right this is a bit weird. Not sure this is not comming from using only one binary feature though. Dec 28, 2021 at 23:40
• Been experiencing exactly the same thing. Accuracy flips from very high to very low. When it's low, what it thinks are the k closest points aren't even remotely close. I've narrowed it down to the choice of the algorithm - choosing "brute" seems to be reliable. Its something I hope to investigate further but I suspect because the other algorithms are approximations they sometimes fail? Apr 2, 2023 at 1:27

$$k$$NN is the most simple supervised learning algorithm: given an input instance $$x$$, it finds the $$k$$ instances most similar to $$x$$ in the training set, and then predicts $$x$$ as the majority class between these $$k$$ instances.
You use a single boolean feature to represent an instance, which means that an instance can only be true or false. The concept of "most similar" has very little sense in this case: a true instance is "equally most similar" to all the training instances which also have true for gender. This means that for every prediction $$k$$NN has to pick the $$k$$ most similar instances randomly among all the instances with the same gender value, and consequently the target class is practically random as well... And this is why the performance is almost entirely determined by the random state.