0
$\begingroup$

Cross-validation

Hi, I'm deploying machine learning models in my MSc thesis using Weka. I have noticed that when I use 10-fold cross-validation in the training dataset I get low evaluation metrics compared to training the model on the entire dataset without cross-validation, this can make sense if I get better performance for the model when I test it on the testing dataset, however, it keeps the same performance. For example, I trained a random forest model for a regression problem and these are the results I have got:

1. Without cross-validation:

Training:

R = 0.97; Mae = 1.31; Rmse = 1.78

Testing:

R = 0.91; Mae = 2.70; Rmse = 3.57

2. With 10-fold cross-validation:

Training:

R = 0.76; Mae = 3.55; Rmse = 4.77

Testing:

R = 0.91; Mae = 2.70; Rmse = 3.57


As you can see the performance of the model on the testing dataset is the same, can someone explain to me what is the purpose of using it if the model is the same in either case?

$\endgroup$
2
$\begingroup$
  • In a short answer, It seems that you used the same hyperparameters for both cross-validation's last training and the simple training process (without cross validation).
  • In the long answer
    • In K-fold cross-validation, you tune the hyperparameters according to the mean of K performance in order to obtain the maximum generalizing ability of the model.
    • Then you train the model last time with the chosen hyperparameters using the whole training set. If you have a test set you use that for the final evaluating of your model.
    • To sum up, the last step of the training part is training the model with the whole training set even if you use k-fold cross-validation or not.
    • So it seems that with cross-validation and without cross-validation you use the same hyperparameters as "max_depth", "minimum sample leaf etc."
$\endgroup$
2
  • $\begingroup$ Yes, I kept the same model hyperparameters, should I use cross-validation with the random forest model? $\endgroup$ – Malek Abozraig Jan 24 at 9:23
  • $\begingroup$ You already used it with a random forest model. You need to tune the hyperparameters to increase the k-fold cv training accuracy, and then probably your test set accuracy will increase either. $\endgroup$ – benan.akca Jan 24 at 18:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.