0
$\begingroup$

I am relatively new in Machine Learning. I am using Random Forest and SVM for a project. Where I did a 10-fold cross-validation on the train set. which gives the following K-Fold Average Score:

Random Forest: 0.8716 & SVM: 0.8665

On the other hand, when I tested with the independent test set it gives the following accuracy:

Random Forest: 93.63% SVM: 90.47%

I am confused is it ok? I mean can the test accuracy be greater than K-fold? Is it what underfitting called. Please help. TIA.

$\endgroup$
  • $\begingroup$ Can you report on the individual fold scores? $\endgroup$ – Ben Reiniger Jul 28 at 2:50
1
$\begingroup$

Absolutely, I've had this happen to me. As Ben Reiniger pointed out in the comment, check your individual scores of the folds. I'd expect you to have some folds that score worse than your reported k-fold score and some that score better, maybe even better than the independent test set.

Some things I consider when this happens:

  • When you test on the independent test set, have you trained on the full training set?

  • What accuracy do you get if you predict on the training set after using it for training? Better or worse than the accuracy you got on the independent test set?

Lastly, consider the balance of classes in your training set, training folds, and your independent test set. Accuracy scores are sensitive to class imbalance, and may cause such a discrepancy in scores between sets if the balance shifts. If you suspect class imbalance is an issue, you could either balance it (e.g. undersample the test set) or use a different metric.

$\endgroup$
0
$\begingroup$

There are two cases:

If K-Fold Average Score is evaluation score average: Then you cant say there is underfitting or or not. You should check training scores average and compare it to unseen test score.

If K-Fold Average Score is training score average: Then you should check evaluation fold score averages.

In both cases, test score can be higher since (I assume) there is no sampling technique you used to split data train/test. You should compare eval scores and training scores in K Folds to interpret overfitting/underfitting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.