1
$\begingroup$

I optimized a knn algorithm in sklearn with a grid search. However, my accuracy on the training data decreased 1% while my cross validation accuracy increased 0.7%. Is the model better after the grid search?

$\endgroup$
1
  • $\begingroup$ Could you share some code? Without any information, it is difficult to know whether the problem comes from the data or the code. $\endgroup$ Commented Aug 15, 2022 at 12:47

4 Answers 4

1
$\begingroup$

Performance on the cross-validation set should be indicative of the performance on out-of-sample data, which is typically what models are ultimately applied to. This is why cross-validation performance is more important and grid search optimizes for it. Better performance on the cross-validation set may come at the expense of worse training set performance. It might be a reason to worry if the performances differ too much, or one drastically increases at the expense of the other, but your numbers seem reasonable, as far as one can tell without knowing more about the use case.

$\endgroup$
1
$\begingroup$

Cross validation accuracy is more important.

Training accuracy is totally meaningless. You should only use it for bug checking.

$\endgroup$
0
$\begingroup$

Your model has low variance in second case then first however the good model is the one which has high accuracy on training set as well as validation data. In other words it is no better than previous

$\endgroup$
0
$\begingroup$
  • In general, the performance on the test set or by cross-validation is the only one which is relevant for evaluation.
  • Note that accuracy is not a good evaluation measure if the dataset is not balanced.
  • kNN is a bit special: there's practically no training, and the testing consists in comparing the test instance against all the training instances.
  • There are very few hyper-parameters to tune in kNN,the main one is the value of k. Thus it's unlikely that there's a lot of improvement to gain with hyper-parameter tuning imho.
  • Depending on the size of the data, a 1% difference might not be significant (i.e. it could be due to chance).
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.