0
$\begingroup$

I've been running K-Fold cross validation multiple times for KNN, random forest and XGBoost.

KNN can complete sklearn's cross_val_score, so much faster consistently.

They all use the same preprocessed data, all have the same test/train split with a random state, etc.

When ran individually the timings are within 10% of each other.

When ran using sklearn's cross val score, KNN can do 50 k-folds in less than 10 minutes, when the other two take over an hour each. (this is consistent, i've ran it 5 times for all 3 algorithms)

Mathematically, or maybe due to sklearn's code, is there a reason for this? (The dataset is 200,000 by 100 columns, at a 25% test/train split)

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

It's not that KNN is better at k-fold cross validation. Its just that KNN doesn't do any training apart from storing a footprint of the training data within the model. KNN's logic resides in its inference step i.e. the predict() call where it determines the k nearest neighbors for the newly supplied instance from previously supplied training data and makes a prediction for the label.

So it's likely that KNN is faster than most other models for small to medium sized datasets.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That makes a lot of sense! Thank you, I didn't think it like that, I wish I could upvote/accept it as an answer but I can't as I'm too low in score. Thank you! If anyone see's this please upvote the answer :) $\endgroup$
    – Denis Vecchiato
    Commented Apr 10, 2021 at 11:41
  • $\begingroup$ You could accept this as the correct answer if you are satisfied with the response. $\endgroup$
    – Jayaram Iyer
    Commented Apr 10, 2021 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.