When to use cross-validation?

Question

Cross-validation

Hi, I'm deploying machine learning models in my MSc thesis using Weka. I have noticed that when I use 10-fold cross-validation in the training dataset I get low evaluation metrics compared to training the model on the entire dataset without cross-validation, this can make sense if I get better performance for the model when I test it on the testing dataset, however, it keeps the same performance. For example, I trained a random forest model for a regression problem and these are the results I have got:

1. Without cross-validation:

Training:

R = 0.97; Mae = 1.31; Rmse = 1.78

Testing:

R = 0.91; Mae = 2.70; Rmse = 3.57

2. With 10-fold cross-validation:

Training:

R = 0.76; Mae = 3.55; Rmse = 4.77

Testing:

R = 0.91; Mae = 2.70; Rmse = 3.57

---

As you can see the performance of the model on the testing dataset is the same, can someone explain to me what is the purpose of using it if the model is the same in either case?

Answer 1

• In a short answer, It seems that you used the same hyperparameters for both cross-validation's last training and the simple training process (without cross validation).
• In the long answer
  • In K-fold cross-validation, you tune the hyperparameters according to the mean of K performance in order to obtain the maximum generalizing ability of the model.
  • Then you train the model last time with the chosen hyperparameters using the whole training set. If you have a test set you use that for the final evaluating of your model.
  • To sum up, the last step of the training part is training the model with the whole training set even if you use k-fold cross-validation or not.
  • So it seems that with cross-validation and without cross-validation you use the same hyperparameters as "max_depth", "minimum sample leaf etc."