How can we apply K-fold cross validation on say linear regression? Regression contains weight update and iterations, and so do we integrate K-fold to it?

If done, is it like we take the first K-1 fold to train using linear regression and perform iterations to obtain weight, then take the next fold and proceed the same and so on until all folds are done and pick the best weight or fold from it?

  • $\begingroup$ If it's a linear model, $\beta$ is given by a famous formula. Why do you bother with iterative weight update? $\endgroup$ – horaceT Mar 13 '18 at 17:01
  • $\begingroup$ If not iterative weight update, how then ? $\endgroup$ – Devi Mar 15 '18 at 11:01
  • $\begingroup$ You need to read up on basic statistics bef jumping onto more advanced subject. A linear model $y = X \beta + \epsilon$, where $X$ is your feature matrix and $\beta$ is your parameter vector, has solution $\hat{\beta} = (X^T X)^{-1} X^{T} y$. Divide your data into 10 portions, calc $\beta$ for 10 separate 9/10th of $X$, predict on the hold-out set, this is 10-fold CV. $\endgroup$ – horaceT Mar 15 '18 at 20:33

How to use cross-validation on regression (assuming 10-fold for example purposes): separate your dataset in 10% and 90%, train on 90%, test your metric (squared error or anything you're modeling) on the remaining 10%. Do that 10 times using different 10% groups. Now you have 10 of your metrics, you can analyze its mean and range to see if the model is robust against over-fitting.

How do you pick a model: you train on the whole dataset. Cross-validation is for testing/validating purposes, you don't use the models generated by cross-validation.

| improve this answer | |
  • $\begingroup$ So, I do a 80-20 split and generate the model, then k-fold for validating its accuracy. Is that so? Please advice. $\endgroup$ – Devi Nov 14 '17 at 5:03
  • $\begingroup$ @Devi Holdout (fixed split) or cross-validation (K-fold split) are validation methods. You usually use one or the other, not both together. $\endgroup$ – Mephy Nov 14 '17 at 15:23
  • $\begingroup$ Yes. But, in percentage split the model is considered which is obtained after training of say 80% records and use that model for remaining 20% test data. Isn't the same procedure valid for k-fold as well exempting that data split into k folds and k-1 fold used for training and remaining for testing (with the model generated from train) and do the same for k different train-test folds thus generating k models ?? $\endgroup$ – Devi Nov 15 '17 at 3:39
  • $\begingroup$ @Devi yes, fixed split and K-fold are similar. But the final model (the one that will be used for predicting/in production) uses all available data, not 80%. The 80/20 split is simply to validate/test it. Validation has nothing to do with the final model. $\endgroup$ – Mephy Nov 15 '17 at 13:27
  • $\begingroup$ There seems a confusion which I want to clarify. I split my i/p data into 80-20 and use 80 data 'only' to train the model and remaining 20 data 'only' to test the generated model to see its accuracy. Later I can give another dataset to the said model and proceed. This is fixed split. $\endgroup$ – Devi Nov 16 '17 at 4:03

Let us assume you have a data set $\mathcal{D}$. We first split this dataset into a training data set $\mathcal{D}_\text{training}$ and a $\mathcal{D}_\text{validation}$. For cross-validation, we only use the training data set $\mathcal{D}_\text{training}$. This data set is divided in $k$-folds $\mathcal{D}_\text{training,1}$, $\mathcal{D}_\text{training,2}$, ..., $\mathcal{D}_\text{training,K}$.

Let us consider the order of the polynomial $M=1,\ldots,M_\text{max}$ as a hyperparameter range.

Set $M=1$: Use $K-1$ of these folds to determine the parameters of the model and validate with the remaining data set. For regression, we often use the mean of the sum of squared errors $\text{MSE}$. As we can do $K$ of these calculations we will obtain $K$ means of the sum of squared errors $\text{MSE}$. Often the mean of these squared errors $\overline{\text{MSE}}_{M=1}$ is used as a measure for the quality of the validation.

Then Set $M=2,...,M_\text{max}$ to obtain $\overline{\text{MSE}}_{M=2}$, ... ,$\overline{\text{MSE}}_{M=M_\text{max}}$.

Finally, pick the value of $M$ for which $\overline{\text{MSE}}$ is minimal. Then use this order to determine the parameters on the full training data set $\mathcal{D}_\text{training}$ and validate on the validations set $\mathcal{D}_\text{validation}$.

In pseudo code/python like

import numpy as np
Initialize dataset $\mathcal{D}$
Initialize the ratio training to validation
Initialize k (the number of folds)
Initialize M_min (the maximum value for the hyperparameter)
Initialize M_max (the maximum value for the hyperparameter)
Initialize array M_values = np.arange(M_min, M_max + 1)
Initialize array MSE_values = np.arange([0] * M_values.size)  
Split the dataset in D_train and D_validation.
Split the training dataset in K folds
for M in M_values:
    Initialize means squared error MSE = 0
    for k in range(1, K + 1):
         Determine data set D_crossval = D_train \ D_train_k
         Determine model for M and D_crossval
         Evaluate model MSE_crossval
         MSE += MSE_crossval
    MSE /= M_values.size 
    MSE_values[M-M_min] = MSE 
M_best = np.argmin(MSE_values) + M_min
print('M_best ={}'.format(M_best))

Determine model for M_best and D_train
Evaluate model on D_validation
| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.