Can we use k fold Cross Validation without any extra (excluded) Test Set?

Question

I have seen this in two Papers:

The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result.

there has been no testing data put aside to validate the final model. These are peer reviewed papers i am talking about. Is this right to do ? Can one even cite such sources?

Answer 1

The key question is

Is K-fold cross validation is used to select the final model (or algorithm)?

1. If yes, as you said, then the final model should be tested on an extra set that has no overlap with the data used in K-fold CV (i.e. a test set).

2. If no, the average score reported from K-fold CV is the final test score, no extra set is required.

Therefore, if authors only picked the best model from all K models, it should have been tested on an extra set. The score of model on the validation fold (from K-fold CV) is not acceptable.

Also, here is a related post on this site (my answer) that goes into more detail about test and validation sets (scores).

EDIT:

I have found a similar question on stats.stackexchange.com. Also, this comment by amoeba suggestions "nested CV" instead of "CV + test set", which I think it is worth fleshing out here.

K-fold CV evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. m[k] = model trained using tr (can be further split into tr2 + v)
  3. score[k] = score of m[k] on ts
3. Test score = average of score[1],...,score[K]

K-fold CV selection and evaluation

1. tr = 80% of data, ts = 20% of data (or some other ratio)
2. For k = [1,..,K]
  1. tr2 = (K-1)/K of tr, v = 1/K of tr
  2. m[k] = model trained using tr2 (can be further split into tr3 + v2)
  3. score[k] = score of m[k] on v
3. M = best of m[1],...,m[K]
4. Test score = score of M on ts

Nested k-fold CV selection and evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. For k2 = [1,..,K2]
     1. tr2 = (K2-1)/K2 of tr, v = 1/K2 of tr
     2. m[k2] = model trained using tr2 (can be further split into tr3 + v2)
     3. score[k2] = score of m[k2] on v
  3. M = best of m[1],...,m[K2]
  4. score[k] = score of M on ts
2. Test score = average of score[1],...,score[K]

Note that in these algorithms, model trained includes parameter learning, hyper-parameter tuning, and model selection (of course, except for the outermost model selection). For example, for selecting between hyper-parameters 8 and 12, we need a deeper loop inside model trained.