Should I repeat the testing/training split while repeating K-fold cross validation?

Question

I'm fitting random forest regressions on my data, and using 10 K-fold cross-validation to evaluate model performance. While re-runing the cross-validation, I noticed that the results differed between each run, sometimes by a lot. So, I decided to repeat the cross-validation calculation 20 times, creating a for-loop, and then summarising the results afterwards. Just to illustrate, I'm doing something like this:

for (i in 1:20) {

trainIndex <- createDataPartition(data$response, p = .8, list = FALSE, times = 1)
data_train <- data[ trainIndex,]
data_test  <- data[-trainIndex,]

train.control <- trainControl(method = "cv", 
                              number = 10)

model <- train(formula, data = data_train, method = "rf",
               trControl = train.control)


pred <- predict(model, data_test)
t<- postResample(pred = pred, obs = data_test$estimate)
t <- t[[2]]

result[[i]] <- t }

So, essentially, each re-run I am splitting my dataset into training and testing sets again. This results in a lot of variance, when ploting the resuls of each run together they vary from ~0.2 to 0.6 R.squared. If I don't do this within the loop (i.e. if I split into training/testing before the loop), then the results of the 20 runs are very similar.

Which way is the right way to go about this?

Answer 1

The purpose of k-fold cross validation is to fit the model multiple times and average out to results to estimate the model's predictive performance.

If having done 10-fold CV once and got one result and then repeating it again and getting a significantly different result, probably means the number of folds is too low, and the k-fold process isn't capturing the inherent variability in the data sufficiently well.

A better test is to change the number of folds and see what happens to the average and variance of the combined output.