I've recently completed the training of a deep learning model for a classification task, using a process that involves k-fold cross-validation for hyperparameter tuning

Initially, I have divided my dataset into 2 parts, ①a training set and ②a separate test set. Then, i divided ①the training set into 5 different folds for hyperparameter tuning. This process helped me identify a set of hyperparameters that maximized the mean accuracy across these 5 different folds.

Then i trained my model, fitted with the best set of hyperparameters with ①the entire training set. Subsequently, i was able to evalute model performance with ②the separate test set, looking at evaluation metrics such as test accuracy, test precision& recall

The problem that i am facing is that I'm unsure how to assess the stability of my model. Typically, one might look at the mean, standard deviation, or confidence intervals of a chosen evaluation metric across different folds or datasets to measure stability. However, given that my final model evaluation is based on ②a single test set (after training with the optimized hyperparameters on ①the entire training set), I'm at a loss for how to present the stability of my DL model in terms of these statistical measures.

Is there an optimal way to calculate or infer the stability of my model under these circumstances? or is it enough to just present a single set of evaluation metrics, with ②a single test set?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.