1
$\begingroup$

I have data for 20 different people and am training a model (e.g. a neural network with the same hyperparameters) on the data from each person; so this gives me 20 models.

I chose to use RMSE to assess the performance. However, since the training data is shuffled, the computed RMSE is nondeterministic and so oscillates. So I thought running each model 10 times and averaging the results, i.e. the RMSE's, would give me a better estimate of performance. But this is for a single person/model. How do I combine the performance of everything, i.e. all 20 models, into a single measure?

Run each of the 20 models 10 times for a total of 200 RMSE values, and average all? Or first compute the average for each person, and then the average of these averages?

Maybe a different method is better? The end goal is to compare a couple of models (e.g. NN vs SVM).

$\endgroup$
1
$\begingroup$

It shouldn't matter if you average all of them or take the average of the average for each person. It turns out that it will give you the same number. This is known as the law of total expectation. Essentially,

E(X)=E(E(X|Y))

Here's the wikipedia article for more details:

https://en.wikipedia.org/wiki/Law_of_total_expectation

|improve this answer|||||
$\endgroup$
  • $\begingroup$ One caveat that I should mention now that I'm coming back to this a few months later, and don't want to be sloppy. I was assuming that you have the same number of models per person. If not, you need to do a weighted average (with weights equal to the number of models for each person) to get the same number as taking the average over all models. $\endgroup$ – Ryan Aug 24 '17 at 19:10
0
$\begingroup$

My two cents
If your intention is training a different model for each individual, then you should consider using more than just average for between model comparison.
Imagine one model performance is great(low RMSE) for 70% of individuals and performs poorly for the other 30% and the second model has mediocre performance across all individuals.
Both models could have the same average but in reality would behave quite differently.
Moreover, you should consider using some sort of statistical measure for the RMSE of each individual model(say a confidence interval of the RMSE). Maybe one model is more volatile than the other.

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.