I have data for 20 different people and am training a model (e.g. a neural network with the same hyperparameters) on the data from each person; so this gives me 20 models.
I chose to use RMSE to assess the performance. However, since the training data is shuffled, the computed RMSE is nondeterministic and so oscillates. So I thought running each model 10 times and averaging the results, i.e. the RMSE's, would give me a better estimate of performance. But this is for a single person/model. How do I combine the performance of everything, i.e. all 20 models, into a single measure?
Run each of the 20 models 10 times for a total of 200 RMSE values, and average all? Or first compute the average for each person, and then the average of these averages?
Maybe a different method is better? The end goal is to compare a couple of models (e.g. NN vs SVM).