3
$\begingroup$

I have seen this in two Papers:

The authors use 10 fold cross validation, and then present the results from this validation or even odder the results from the best Fold as their modelling Result.

there has been no testing data put aside to validate the final model. These are peer reviewed papers i am talking about. Is this right to do ? Can one even cite such sources?

$\endgroup$
3
$\begingroup$

The key question is

Is K-fold cross validation is used to select the final model (or algorithm)?

  1. If yes, as you said, then the final model should be tested on an extra set that has no overlap with the data used in K-fold CV (i.e. a test set).

  2. If no, the average score reported from K-fold CV is the final test score, no extra set is required.

Therefore, if authors only picked the best model from all K models, it should have been tested on an extra set. The score of model on the validation fold (from K-fold CV) is not acceptable.

Also, here is a related post on this site (my answer) that goes into more detail about test and validation sets (scores).

EDIT:

I have found a similar question on stats.stackexchange.com. Also, this comment by amoeba suggestions "nested CV" instead of "CV + test set", which I think it is worth fleshing out here.

K-fold CV evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. m[k] = model trained using tr (can be further split into tr2 + v)
  3. score[k] = score of m[k] on ts
3. Test score = average of score[1],...,score[K]

K-fold CV selection and evaluation

1. tr = 80% of data, ts = 20% of data (or some other ratio)
2. For k = [1,..,K]
  1. tr2 = (K-1)/K of tr, v = 1/K of tr
  2. m[k] = model trained using tr2 (can be further split into tr3 + v2)
  3. score[k] = score of m[k] on v
3. M = best of m[1],...,m[K]
4. Test score = score of M on ts

Nested k-fold CV selection and evaluation

1. For k = [1,..,K]
  1. tr = (K-1)/K of data, ts = 1/K of data
  2. For k2 = [1,..,K2]
     1. tr2 = (K2-1)/K2 of tr, v = 1/K2 of tr
     2. m[k2] = model trained using tr2 (can be further split into tr3 + v2)
     3. score[k2] = score of m[k2] on v
  3. M = best of m[1],...,m[K2]
  4. score[k] = score of M on ts
2. Test score = average of score[1],...,score[K]

Note that in these algorithms, model trained includes parameter learning, hyper-parameter tuning, and model selection (of course, except for the outermost model selection). For example, for selecting between hyper-parameters 8 and 12, we need a deeper loop inside model trained.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.