Let us consider a case where $1000$ data is given, i.e., the data set $U=\{x_1, \ldots, x_{1000}\}.$

When we want to use $k$-fold validation scheme, we first divide the data set into $k$ groups.

With out loss of generality, the parameter $k$ is assumed to be $10$.

Hence, we have $S_1=\{x_1, \ldots, x_{100}\}$, $S_2=\{x_{101}, \ldots, x_{200}\}$, $\ldots$, $S_{10}=\{x_{901}, \ldots, x_{1000}\}$.

I can obtain models, $f_k$, by learning a data set $U \setminus S_k$ for $k=1, 2, \ldots, 10$.

I can obtain error rates, $r_k$, by testing a data set $S_k$ with $f_k$.

Hence, I can obtain the error rates, $r$, by averaging $r_k$'s, i.e., $\sum_{k=1}^{10}r_k/10$.

I understand the $k$-fold cross validation so far.

Most materials I've seen just say the error rate averaged by $k$ scenarios in $k$-fold cross validation. However, they do no t say about $f_k$'s.

However, in this case, which model do I have to use among all $f_k$'s?

  • $\begingroup$ The better way out is to average all your models test set predictions and return it... In order to get all the models, it's simple enough that you append the model name itself as a list, it will store the (example) Booster object address which we can then address by list indexing ...(+1 Nice Question..) $\endgroup$
    – Aditya
    Jun 2 '18 at 1:51

$k$-fold is just performed to obtain a measure of your accuracy, as using the training accuracy is generally a too optimistic measure of the accuracy. If you want to deploy a final model, what is recommended is to train a last model with all the data. In fact, when you compare two models, $f$ and $g$, what you do is obtain two error rates $r_f$ and $r_g$ by cross-validation, and you keep the model with the lowest error rate. After that, if the model with the lowest error rate is $f$, you retrain $f$ with all your data.

To sum up, $k$-fold cross-validation is a method to give measures of the performance of the model, if you want to obtain the best model just train it with all your data.

  • $\begingroup$ Oh, I see. So far, I have misunderstood the concept of the cross validation. Thank you. $\endgroup$
    – Danny_Kim
    Jun 1 '18 at 10:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.