2
$\begingroup$

I was performing a binary classification problem with 15000 RGB images using a scratch build CNN model. While it comes to evaluate the model, I can do it in two ways:

  1. Splitting data Train and Test and use 10 fold cross-validation for the training data. Later with the best model, I would use the unseen Test data. In this way I got appx. 91.5% avg. accuracy for both test and validation.
  2. Just use 10 fold cross-validation and got 92.5% avg accuracy(slightly better result than the previous one.)

Which option would be the best for reporting the performance of my model in the research article? TIA

$\endgroup$
4
  • 1
    $\begingroup$ sorry I started writing an answer but suddenly I realized that there's a problem: in option 1 your best model with CV gives 91.5% on its own validation set? If yes that's strange since it should be at least as good as your average performance in option 2, since it's supposed to be the best model. Anyway the short answer is to use your option 2, that's the proper way to evaluate with CV (your option 1 is not proper CV, I'll explain later) $\endgroup$
    – Erwan
    Sep 9, 2020 at 22:57
  • $\begingroup$ Can you explain a bit. It would be great help. Thank you. $\endgroup$ Sep 10, 2020 at 11:57
  • $\begingroup$ What I don't understand is when you say in option 1 that you get 91.5% average accuracy for both test set and validation set: as far as I understand you apply the best model out of the 10 CV runs on the unseen test set, right? If yes what is the validation set in this case, is it the CV inner test set for this run (i.e. the 10% of the training set used as test set)? Or is 10xAI correct to assume that you use CV to tune some hyper-parameters? But in this case how do you select the parameters in option 2? $\endgroup$
    – Erwan
    Sep 10, 2020 at 13:00
  • $\begingroup$ In case of the best model, The model which achieved best accuracy(i.e. 93%) in validation set while in 10-fold CV. I use that model for to test unseen test data and got accuracy 91.6% while the average accuracy of 10 fold is 91.5%. $\endgroup$ Sep 10, 2020 at 13:38

2 Answers 2

2
$\begingroup$

In my point of view, it should be Option-I
Primary logic is to test your Model on unseen data.

Reason for that is,
When you executed your K-Fold and let's say got a score ~87% and then you tweaked your Hyperparameters.
In this way, we actually leak the test data(K-Fold test set) information to the learning process, and eventually, the process will overfit to the test data too if this hit-try happens multiple times.

So, the Option-I score should be closer to the score on future new data.

$\endgroup$
1
$\begingroup$

I'm still not 100% sure about the setting, but based on OP's comments I understand that there is no hyperparameter tuning, so there's a single method being trained in two different ways. So if my understanding is correct:

  • In option 1 the training data is used for CV training/testing, then the model which corresponds to the best CV run is selected and applied to the unseen test set. This would be an unusual way to use CV, since normally CV is used only for evaluation, not for extracting one of the models. Unsurprisingly the performance of the model on the unseen test data is lower than during CV, because the maximum performance during CV is likely due to chance.
  • Option 2 is just regular CV evaluation for a single model, so I would use this result.

However there is an inconsistency between the results obtained: if in option 1 the average CV accuracy is 91.5, there's no logical reason why it's 92.5 in option 2 (there's slightly more data but it's unlikely to improve that much).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.