Best way to implement 10-fold cross validation for CNN model

Question

I was performing a binary classification problem with 15000 RGB images using a scratch build CNN model. While it comes to evaluate the model, I can do it in two ways:

1. Splitting data Train and Test and use 10 fold cross-validation for the training data. Later with the best model, I would use the unseen Test data. In this way I got appx. 91.5% avg. accuracy for both test and validation.
2. Just use 10 fold cross-validation and got 92.5% avg accuracy(slightly better result than the previous one.)

Which option would be the best for reporting the performance of my model in the research article? TIA

Answer 1

In my point of view, it should be Option-I
Primary logic is to test your Model on unseen data.

Reason for that is,
When you executed your K-Fold and let's say got a score ~87% and then you tweaked your Hyperparameters.
In this way, we actually leak the test data(K-Fold test set) information to the learning process, and eventually, the process will overfit to the test data too if this hit-try happens multiple times.

So, the Option-I score should be closer to the score on future new data.

Answer 2

I'm still not 100% sure about the setting, but based on OP's comments I understand that there is no hyperparameter tuning, so there's a single method being trained in two different ways. So if my understanding is correct:

• In option 1 the training data is used for CV training/testing, then the model which corresponds to the best CV run is selected and applied to the unseen test set. This would be an unusual way to use CV, since normally CV is used only for evaluation, not for extracting one of the models. Unsurprisingly the performance of the model on the unseen test data is lower than during CV, because the maximum performance during CV is likely due to chance.
• Option 2 is just regular CV evaluation for a single model, so I would use this result.

However there is an inconsistency between the results obtained: if in option 1 the average CV accuracy is 91.5, there's no logical reason why it's 92.5 in option 2 (there's slightly more data but it's unlikely to improve that much).