Multiple Linear Regression with k-fold Cross Validation

Question

I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?

Answer 1

As Peter said, you need to split your dataset into two subsets: Training, and Test sets. Generally, 80% of data is allocated for Training set (20% for the Test set). Thereafter, depending on the language/package you use (caret in your case), you use 5- or 10-fold cross-validation to train your model, and finally, you check the prediction ability of the model using the Test set.

I quickly checked the Caret Package website and ripped the required code for you.

1- For Training-Test split:

4.1 Simple Splitting Based on the Outcome

library(caret)
set.seed(3456)
trainIndex <- createDataPartition(iris$Species, p = .8, 
                                  list = FALSE, 
                                  times = 1)

2- For training with 10-fold cross-validation:

5.3 Basic Parameter Tuning, 5.5.4 The trainControl Function

fitControl <- trainControl(## 10-fold CV
                       method = "repeatedcv",
                       number = 10,
                       ## repeated the CV ten times
                       repeats = 10)

Usually, we do the cross-validation only once (repeats = 1); but to check the consitency of the results, you may need more repeats.