0
$\begingroup$

I am training models with a small dataset (around 800 observations) and I am using Repeated K-Fold cross validation to evaluate the models.

Initially, i am using the same cross validation for hyperparameter-tuning, and then when i have found the optimal hyperparameters, i am training a new model with those parameters using Repeated K-Fold Cross Validation. Do i still need to use a seperate Test set to evaluate the model after all of this, or are the evaluations from the Repeated K-Fold Cross Validation sufficient?

I am also worried that if i do need a seperate test set, it will be very small (<80 observations), so may not provide reliable results, is this a real cause for concern?

Thanks in advance.

$\endgroup$

1 Answer 1

1
$\begingroup$

A few short comments:

  • "small" is relative and depends on the complexity of the model to be learned
  • I don't understand how exactly you train your model after you found the best hyperparameter, could you clarify?
  • in any case, you always have to evaluate the model performance based on a separate test dataset

Here are the answers to the additional information/questions:

After finding the best hyperparameter for the model, I then define a new model using those parameters and train that model with repeated k-fold cross validation, is this correct?

There is more than just one workflow for hyperparameter optimization and training/testing. In your case, your first run of k-fold cross validation (CV) is only used for determining hyperparameter. There, you use the held-out dataset to evaluate the performance of the candidate models. Once, you knwo the performance of all candidate models, you can choose the best-performing model, the so-called "final model". A strategy is, e.g., to use the candidate model that during CV gives -- on average -- the best performance.

The next step is to train the final model, e.g., for the purpose of prediction. Again, you have a number of options for evaluating the "goodness" of the training but all of them involve that you kept a dataset, the testing dataset, which has not been used in the process before. However, you can use everything MINUS the testing dataset (i.e., the previous training and also the testing folds) as training dataset for training the final model. For (cross) validation of the training process you can, as always choose between the hold-out method (where you split the whole dataset into a single training and validation set) or, e.g., the k-fold CV.

Regarding the separate test dataset, the model gives different results depending on the train test split, i assume this is due to how small the test set is, how can i mitigate this? Should i train the model on multiple train test splits?

This touches upon another subject: the sample and the population. Ideally, the training, validation or testing datasets should be representative for the population (i.e., the entire possible dataset) but most of the time we only have a small subset available, the sample. If the sample (e.g., the testing dataset) is large enough then we can use it for statistical analysis and use it to guess something about the whole population. The nice thing about k-fold CV is that this gives you a way to see if the folds are statistically representative. In your case, the dataset seems to be rather small indeed. There are several options three of which are: 1. try to analyze the dataset and see if it has a strong imbalance -- this might then require some (re)sampling stragey; 2. choose the value of k carefully as it determines the size of the folds (you read about k=5 or 6 is a good value but in your case you might want to choose a lower number); 3. get more data. In any case, training the model on multiple runs of train-test splits should not be done as in this case you would be using testing data that already has been used for training.

$\endgroup$
5
  • $\begingroup$ Thanks for the response! After finding the best hyperparameters for the model, i then define a new model using those parameters and train that model with repeated k-fold cross validation, is this correct? Regarding the separate test dataset, the model gives different results depending on the train test split, i assume this is due to how small the test set is, how can i mitigate this? Should i train the model on multiple train test splits? Thanks in advance. $\endgroup$ Oct 9, 2023 at 11:14
  • $\begingroup$ There are a number or different aspects: 1. clearly separate the hyperparameter optimization from training a "final" model. 2. try to understand if you split data is statistically representative. I've added details of that to my previous answer. As you will see, this is not an entirely trivial problem... $\endgroup$
    – BanDoP
    Oct 9, 2023 at 13:50
  • $\begingroup$ Great, thank you so much for the response this is very helpful. Touching on a few points you made, the dataset is indeed imbalanced, so i have upsampled the training set using SMOTE to balance the outcome variables, is it wise to do this considering the small nature of the testing set i.e. will the model be susceptible to over predicting the minority case? Furthermore, using k-fold cv for the 'final model' yields great results for the metrics i am using, however when tested on the test set the results are significantly lower, could this be a sign of overfitting? $\endgroup$ Oct 9, 2023 at 14:01
  • $\begingroup$ You're already doing a number of correctly. And yes, this might be a sign of overfitting. Don't forget to "look at the data" (directly or via visualization) to get a hint of what is going on there. Everything else would probably be a different question ;) $\endgroup$
    – BanDoP
    Oct 9, 2023 at 14:12
  • $\begingroup$ Okay i will have a look. Yeah you're right, got carried away with all my problems, thank you for all the help its really appreciated :) $\endgroup$ Oct 9, 2023 at 14:21

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.