What is the meaning of K-folding Cross Validation's mean error?

Question

I was just wondering the meaning of getting mean error of k-folding Cross Validation.

The process is when I split a data set into k folding, and using k-1 as training set and the left 1 as the validation set, And then I train on the k-1 pieces of data, I got model M1;

Then I continue from M1, and train the model based on the same split set(take another 1 piece as validation set while the rest k-1 as training set); then train, get model M2;

... (we have k ways to switch the validation/training sets,so k rounds of training)

Based on the process; we get model from M1 to Mk;

The mean error of the k-folding is defined by (error(M1)+...error(Mk))/k; However, since we have improved/advanced in each round's training, if I want to use the final Mk as our model, why we use mean error rather than error on Mk to estimate the error.

Or I misunderstood the process, Mk is not the best model we will use, instead, we use the vote/mean or combination of all the M1...Mk to make the prediction? Each model is just another part of the whole picture, we didn't treat Mk as the most advanced version of all?

Or no matter we use M1...Mk or just Mk; the mean of the k-folding error is always a good estimation?

Answer 1

When you perform $k$-fold cross-validation you obtain $k$ models, but each $M_k$ is independently trained and so you don't start training from the previous model $M_{k-1}$: sequential training happens in boosting ensemble (like AdaBoost), which is unrelated to $k$-fold cross-val.

Training various models on different subset of the dataset allows to estimate the bias-variance of them: you get $k$ models which implicitly define a distribution over their performance. If you aggregare them (i.e. take the mean of their accuracy, for example) you get an estimate of the average performance of the class of models (e.g. SVM + chosen hyper-parameters) that you've used. Also, you can compute the variance in both predictions and accuracy: that provides you a means to estimate the uncertainty of your models.

Indeed, you obtain $k$ models. So, you can either pick the best of them (e.g. the one that achieves the best accuracy) or aggregate them in an ensemble.