0
$\begingroup$

I'm fitting random forest regressions on my data, and using 10 K-fold cross-validation to evaluate model performance. While re-runing the cross-validation, I noticed that the results differed between each run, sometimes by a lot. So, I decided to repeat the cross-validation calculation 20 times, creating a for-loop, and then summarising the results afterwards. Just to illustrate, I'm doing something like this:

for (i in 1:20) {

trainIndex <- createDataPartition(data$response, p = .8, list = FALSE, times = 1)
data_train <- data[ trainIndex,]
data_test  <- data[-trainIndex,]

train.control <- trainControl(method = "cv", 
                              number = 10)

model <- train(formula, data = data_train, method = "rf",
               trControl = train.control)


pred <- predict(model, data_test)
t<- postResample(pred = pred, obs = data_test$estimate)
t <- t[[2]]

result[[i]] <- t }

So, essentially, each re-run I am splitting my dataset into training and testing sets again. This results in a lot of variance, when ploting the resuls of each run together they vary from ~0.2 to 0.6 R.squared. If I don't do this within the loop (i.e. if I split into training/testing before the loop), then the results of the 20 runs are very similar.

Which way is the right way to go about this?

$\endgroup$
0
$\begingroup$

The purpose of k-fold cross validation is to fit the model multiple times and average out to results to estimate the model's predictive performance.

If having done 10-fold CV once and got one result and then repeating it again and getting a significantly different result, probably means the number of folds is too low, and the k-fold process isn't capturing the inherent variability in the data sufficiently well.

A better test is to change the number of folds and see what happens to the average and variance of the combined output.

$\endgroup$
1
  • $\begingroup$ Thank you for your reply. I think the issue is not with the CV itself, but with the training/testing split. When I keep the training/testing split the same, the CV results don't differ all that much between runs. But when I re-run the training/testing split, then the CV results vary a lot between runs. I'm not quite sure how to interpret this or what to do with it! $\endgroup$ May 16 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.