2
$\begingroup$

In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.

Source : https://scikit-learn.org/stable/modules/cross_validation.html

However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.

Thanks

$\endgroup$
3
$\begingroup$

It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).

But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.

(this is my interpretation, there might be other reasons)

$\endgroup$
1
$\begingroup$

I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.

Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).

$\endgroup$
0
$\begingroup$

Purpose of K-Fold cross validation is to evaluate new model performance on unseen data only.

So how K-fold works?

  1. Model gets trained with training data.
  2. Split the cross validation data set in to K-Folds(or K equal parts).
  3. Get the hyper-parameter from the step-1 and train a new model with K-1 folds.
  4. Evaluate step-3 model using left over Kth fold.
  5. Repeat the process K times.
  6. Take mean value of accuracy.

So we need a score of all K folds for model evaluation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.