I have to compare several deep learning models (CNNs) based on the same dataset.
For estimating the model skill's I use the
train_test_split instead of
k-fold cross validation, because I have a lot of test data (10000 samples).
But I have a problem with estimating the stochastic model skill's (Model Stability). The problem is the randomness of deep learning models. I read about a recommendation to train each model at least 30 times and then calculate the mean, the so-called grand mean. This way would give a robust estimate.
But I have more than 20 models to compare and it would be very time consuming to run every model 30 times.
What are your suggestions, opinions and experiences when comparing several deep learning models?
Thanks in advance.