0
$\begingroup$

I read that we need cross-validation in KNN algorithm as the K value that we have found from the TRAIN-TEST of KNN might not be generalizable on unseen data.

The logic given was that, the TEST data set was used in finding K value, and thus the KNN-ALGORITHM is having information of the TEST dataset because K was found by using TEST dataset. And it is not same as unseen data.

But YES, the K value was found from the TEST dataset and thus we get our KNN-Algorithm, but the testing on TEST data was carried out without knowledge of TEST data and and yes we picked K from that but irrespective of our picking that k or not, KNN was giving that accuracy on TEST data that was blind.

So why need for cross-validation because of this?

$\endgroup$
2
  • $\begingroup$ Could you rewrite the last part. I'm having trouble understanding it.You only used TEST dataset in this part and it feels like you meant TRAINING dataset in some parts. $\endgroup$
    – RSale
    Jan 9, 2022 at 14:03
  • $\begingroup$ @RSale I meant "TRAINING" data to produce a KNN classifier i.e. just store the training dataset in RAM. and then plot a "Accuracy vs K plot" on TEST dataset to get best K $\endgroup$ Jan 10, 2022 at 9:16

3 Answers 3

1
$\begingroup$

It is a best practice to use crossvalidation to find the best hyperparameter values.

If you test several different hyperparameter values (like some different K values for KNN) it is custom to have:

  • a training data set:

    The train data set is split in k folds to apply crossvalidation

    You do crossvalidation and get the metric with different values of k

    Then you select the best k for you final model

  • a testing data set:

    With all the model and hyperparameters chosen you do a final measure on the test set to check if your model and hyperparameters are good on unseen data

This way you are sure to keep some unseen data for the final model evaluation, and avoid data leakage while choosing and selecting the best hyperparameter value k.

$\endgroup$
2
  • $\begingroup$ Yes the procedure I understand. But the "cross-validation step" when we find the best k. That "Accuracy vs K" plot is on unknown data where what we learnt is the TRAINING DATA. Thus the best k, for example k=3 gives me 90% accuracy and I pick that. So, why do I need to again have a unseen "TEST" data to measure my accuracy? Why can't I take that 90% as my accuracy? Thanks for the answer $\endgroup$ Jan 10, 2022 at 9:19
  • $\begingroup$ This is the principle of having a train set and a test set. You establish your model using only training data set. Cross validation and selecting a k is part of building and training your model. So the selection of k may be biased. So you need to asses again the accuracy with a totally untouched/unused data set : this is your final test set. With the test set accuracy, you can compare and tell if model is over/underfitting or good enough. $\endgroup$
    – Malo
    Jan 10, 2022 at 10:38
1
$\begingroup$

In addition to what Malo said.

Cross validation actually solves another problem. We used to split the data into 3 sets. A training set to fit the model, a test set to fine tune the parameters and a validation set for the final test. If you do this split only once then the model learns only with the training set provided. So the learning depends on how you decide to split the dataset. The dataset to learn with gets very small. With cross validation you split the dataset multiple times and you learn and fine tune on every k-fold. With every iteration you might get new information from the dataset and you increase the data you can use for training and fine tuning. In the graph below you can think of the orange test data as validation data.

https://scikit-learn.org/stable/modules/cross_validation.html(from https://scikit-learn.org/stable/modules/cross_validation.html )

If you want to test a pupil you'll make sure that the pupil doesn't have access to the exam answers, right? You want to know if the pupil can perform well on an exam which he/she hasn't seen before. It's the same thing here. If you train and fine tune a model on a dataset you want to make sure that it performs well on a new unseen test set.

Let's stay with the pupil analogy here. Let's assume each row of the matrix in the graph is a week of learning for a final exam. The final exam is the orange test data. Every week the pupil studies with the study materials (green) and does a training test (blue). The next week the study material changes slightly and the pupil has a new training test (blue). Once all weeks of learning are complete the pupil takes the final exam(orange) which has never seen before questions.

See this post for a detailed explanation difference-test-validation-datasets/

$\endgroup$
0
$\begingroup$

and yes we picked K from that but irrespective of our picking that k or not, KNN was giving that accuracy on TEST data that was blind.

Yes, and this is why the selection process isn't completely useless; we picked the hyperparameter(s) based on performance on unseen data.

But that selection still introduces bias to the score. To see that, pretend that your hyperparameter actually does nothing, so that the scores on the test folds vary only due to randomness in your model fitting procedure (OK, so something other than kNN). Then the test scores are just random numbers with some distribution around the true/asymptotic performance. The selection method selects the hyperparameter displaying the maximum value from this random sample. That's obviously higher than the true expected performance, which lies somewhere around the mean of your sample values.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.