# Multiple Linear Regression with k-fold Cross Validation

I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?

• If your ultimate aim is prediction, I would set aside some data to test performance after all steps described by you. For training (model selection) you still can use 5-fold CV on the training set. You also can check assumptions based on the full trainings set. Test data should not be used in model selection/training. Commented Dec 8, 2019 at 19:21

As Peter said, you need to split your dataset into two subsets: Training, and Test sets. Generally, 80% of data is allocated for Training set (20% for the Test set). Thereafter, depending on the language/package you use (caret in your case), you use 5- or 10-fold cross-validation to train your model, and finally, you check the prediction ability of the model using the Test set.

I quickly checked the Caret Package website and ripped the required code for you.

1- For Training-Test split:

4.1 Simple Splitting Based on the Outcome

library(caret)
set.seed(3456)
trainIndex <- createDataPartition(iris\$Species, p = .8,
list = FALSE,
times = 1)


2- For training with 10-fold cross-validation:

fitControl <- trainControl(## 10-fold CV
method = "repeatedcv",
number = 10,
## repeated the CV ten times
repeats = 10)


Usually, we do the cross-validation only once (repeats = 1); but to check the consitency of the results, you may need more repeats.