0
$\begingroup$

I would first like to create few multiple regression models based on if the models violate any multiple regression assumptions and how well it fits the training data. Then I would like to compare how well these models predict new observations by using 5-fold Cross Validation. From my understanding 5-fold Cross Validation shuffles then splits my data into 5 groups and chooses 1 for the testing set, and the other 4 for the training set. A given model is tested and the prediction error is recorded. This is repeated until all 5 groups are used as a testing set. Finally, the prediction errors are averaged.

My question is, when I am first determining the multiple regression equation (checking for assumptions, applying transformations, variable selection, etc.) which set of data should I use as my training set? Do I use the entire data set? Do I use one of the 5 training sets created by the 5-folds CV method? Do I repeatedly try to fit the regression model for all 5 training sets? If so, how would I extract each training set using the caret package?

$\endgroup$
1
  • 1
    $\begingroup$ If your ultimate aim is prediction, I would set aside some data to test performance after all steps described by you. For training (model selection) you still can use 5-fold CV on the training set. You also can check assumptions based on the full trainings set. Test data should not be used in model selection/training. $\endgroup$
    – Peter
    Dec 8, 2019 at 19:21

1 Answer 1

1
$\begingroup$

As Peter said, you need to split your dataset into two subsets: Training, and Test sets. Generally, 80% of data is allocated for Training set (20% for the Test set). Thereafter, depending on the language/package you use (caret in your case), you use 5- or 10-fold cross-validation to train your model, and finally, you check the prediction ability of the model using the Test set.

I quickly checked the Caret Package website and ripped the required code for you.

1- For Training-Test split:

4.1 Simple Splitting Based on the Outcome

library(caret)
set.seed(3456)
trainIndex <- createDataPartition(iris$Species, p = .8, 
                                  list = FALSE, 
                                  times = 1)

2- For training with 10-fold cross-validation:

5.3 Basic Parameter Tuning, 5.5.4 The trainControl Function

fitControl <- trainControl(## 10-fold CV
                       method = "repeatedcv",
                       number = 10,
                       ## repeated the CV ten times
                       repeats = 10)

Usually, we do the cross-validation only once (repeats = 1); but to check the consitency of the results, you may need more repeats.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.