In k-fold-cross-validation, the "correct" scheme seem to compute the metric (say the accuracy) for each fold, and then return the mean as the final metric.
Source : https://scikit-learn.org/stable/modules/cross_validation.html
However, why can't we just compute the metric directly on all the predictions, as we have a estimation for all the data.
Thanks
It's ok to compute the global performance on the concatenation of the predictions for all the K folds after running the cross-validation process, it depends on the goal and on the metric (for instance the mean accuracy over the folds gives the same result as the global accuracy, but that's not true for every evaluation measure).
But very often the goal involves not only measuring performance accurately but also measuring the variance of the performance across the folds, in order to detect instability. This can't be done from the concatenation of the predictions, so it's often more convenient to keep the folds results separated.
(this is my interpretation, there might be other reasons)
I'm not 100% sure what you mean. In k-fold CV, you partition the training set into $k$ subsets of equal size. Holding out one of these folds at a time, you train the model on the remaining $k-1$ folds to make a prediction for the held-out fold. Thus, in the end, you have one prediction for each observation in your training data.
Now, you can compute average accuracy in two equivalent ways: for each fold, compute the average accuracy, then average the k averages. Or, you average accuracy for every single observation. The two values are the same (up to rounding errors), since all you do is take the intermediate step of calculating averages for each fold. The advantage is that you save on memory because you only have to retain $k$ values (average accuracy per fold) instead of $N$ values (one value for each observation in the training set).
Results can be different (even if in practice, it does not really maters, but only for some metrics!)
Let's say you perform a 2-fold cross validation on a set with 11 observations. So you will have an iteration with a test set with 5 elements, and then another with 6 elements.
If you compute the compute the accuracy globally, thanks to a global confusion matrix (which will have 5+6=11 elements), that could be different than computing the mean from the two folds. Because, with the mean procedure, you will put the same weight (here =0.5) to every folds, even if they do not have the exact same number of observations. While, with the global procedure, you will, kind of, put different weights to each observations.
Of course, when you have more observations than 11 (...), the difference is insignificant (but is still present). And if you have the (exact) same number of observations in each fold, the mean procedure gives a result equals to the global one (for example for a 2-fold CV with 10 and 11 elements).
But for other metrics, for example for the precision of a given class, that raises a more fundamental question, even if in practice it is still rare.
Let's say you perform a 5-fold CV for a classification task, but one of the target class is not so much present. The model sometimes, for a certain iteration of the 5-fold but not for all, could learn not to predict that class at all. So, mathematically, you can't compute the precision for that given modality, since your model does not predict any test observation as belonging to that class.
For example, for 3 of the 5 folds the models does, and for the other 2 it does not. If you compute a global confusion matrix, then compute, from it, the precision from that class, you, mathematically, can do it. But if you do the "mean/average procedure", you can't since, for some (2) of the folds, you were not able (mathematically) to compute it.
So, and it could be another question :), which one (global or mean) should we choose for computation of metrics that sometimes can't be computed. Of course you could define, so decide, a value to allow every time the metrics can't be computed (let's say 0), but it is not really the debate here...)
Here is a link to a certain article (not the only one) that discusses your question (if link is broken the article name is "Apples-to-Apples in Cross-Validation Studies: Pitfalls in Classifier Performance Measurement" from Forman and Scholz).
