3
$\begingroup$

K-fold cross-validation divides the data into k bins and each time uses k-1 bins for training and 1 bin for testing. The performance is measured as the average across all the K runs err ← err + (y[i] − y_out)^2 as demonstrated in Wikipedia and the literature

   err ← 0
   for i ← 1, ..., N do
    // define the cross-validation subsets
     x_in ← (x[1], ..., x[i − 1], x[i + 1], ..., x[N])
     y_in ← (y[1], ..., y[i − 1], y[i + 1], ..., y[N])
     x_out ← x[i]
     y_out ← interpolate(x_in, y_in, x_out)
     err ← err + (y[i] − y_out)^2 
   end for
   err ← err/N

But what about the parameters that are obtained from the training? is it the average across all the training or does it to be picked from the best output in k-fold cross-validation? Do we need to run the same ML algorithm in k-fold cross-validation or each fold can have a different algorithm? I think we need to run only one algorithm for the k-fold and for each individual algorithms we need to run k-fold cross-validation.

Improve this question
$\endgroup$
2
  • 1
    $\begingroup$ What are you trying to do with the cross validation? A common use is to optimize hyperparameters, such as the penalty in regularized regression. The approach would be to try out many different regularization penalties on the $k$ folds, then pick the best one according to the five out-of-sample performance calculations. Then you go and fit on the whole data set (except maybe a final holdout sample). $\endgroup$
    – Dave
    Commented Jul 26, 2021 at 20:53
  • $\begingroup$ I am trying to understand how it works, So these are the hyperparameters of a single algorithm or each time a new algorithm will be executed? I cannot understand "then pock the best one according to the five out-of-sample performance calculations", what do you mean? does it pick the best hyperparameter among then k different execution? if yes, then why is the error the average of the K executions? $\endgroup$
    – Mohsen Sichani
    Commented Jul 26, 2021 at 21:15

1 Answer 1

Reset to default
5
$\begingroup$

Cross-validation is a method to obtain to obtain a reliable estimation of the performance. The performance is obtained as the average across the CV "folds" because this way it doesn't depend on a single test set, i.e. the impact of chance is minimized.

In the case of hyper-parameter selection, the goal is not only to evaluate but also to select the hyper-parameters values based on this evaluation. This turns the CV process into a training stage, because it is used to determine something about the model.

When the goal is to select the best hyper-parameters among a set of possible assignments of their values, the method is run across all the CV "folds" for every possible assignment, and then the average performance is also obtained for every assignment. At the end of the CV process the assignment which corresponds to the maximum average performance is selected.

Now that the parameters are fixed, one still has to determine the true performance on a fresh test set because the high performance among the parameters assignments could be due to chance. This is why a model is trained again with these parameters (usually using the whole training data), then applied to a fresh test set to obtain the final performance.

Notice that everything in CV is done the same way across the "folds": the same method(s) are run for every fold, and the results are always obtained across all "folds". In particular one should never select the best model or parameters by picking the maximum "fold".

Improve this answer
$\endgroup$
8
  • $\begingroup$ Thanks @Erwan for the explanation, just a couple of questions, assume a 3-fold CV, in the first run (first fold) the slope is 2 (slope 2 gives the minimum of MSE), second run (second fold) slope is 3 and it gives the min MSE, and for the third run the slope is 4 (with min MSE), so the slope that appears as the result of this linear regression will be 3 ((2+3+4)/3) , right? does the algorithm will be run one more time on the whole dataset or 3 is the maximum number of runs? $\endgroup$
    – Mohsen Sichani
    Commented Jul 27, 2021 at 3:21
  • 2
    $\begingroup$ @MohsenSichani no, the parameters themselves are not averaged. You might have a confusion between regular model parameters and hyper-parameters: the slope in a linear model is a regular parameter, as opposed to for instance the value of $k$ in $k$-nn. The difference is that the values of the hyper-parameters are not determined by the training (as opposed to regular model parameters), they must be decided another way and hyper-parameter tuning with CV is a ... $\endgroup$
    – Erwan
    Commented Jul 27, 2021 at 10:16
  • $\begingroup$ ... common method. Now to answer your question about the slope in a linear model: CV is used only for evaluation, not for actually producing the final model with its parameters. The final model (i.e. the values of the regular parameters) should simply be trained on the whole training set, it's not directly related to using CV. $\endgroup$
    – Erwan
    Commented Jul 27, 2021 at 10:20
  • 1
    $\begingroup$ @MohsenSichani: CV is only meant to reliably evaluate a method, not to train the final model. It's a common confusion because it seems logical that since training happens during CV (k times), one could use one of the trained models as final model. But (1) this does not use the full data and (2) it doesn't make sense to pick the best model because this defeats the purpose of CV. the literature says that k-1 are used for training inside the CV process, it doesn't say that any of the CV models should be used as final model. $\endgroup$
    – Erwan
    Commented Jul 28, 2021 at 16:40
  • 2
    $\begingroup$ Note that CV is not always needed, the standard way to train and test a model is to randomly split the data between a training set and a test set, train the model on the training set and evaluate on the test set. CV offers a slightly more reliable measure of performance but that's not always needed. And yes, it's used for tuning hyper-parameters for the same reason. $\endgroup$
    – Erwan
    Commented Jul 28, 2021 at 16:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.