Suppose that we have train a model (as defined by its hyperparameters) and we evaluated it on a test set using some performance metric (say $R^2$). If we now train the same model (as defined by its hyperparameters) on a different training data we will get (probably) a different value for $R^2$.
If $R^2$ depends on the training set, then we will obtain a normal distribution around a mean value for $R^2$. Shouldn't therefore average the $R^2$ from the various evaluations in order to get a better picture of the models performance? Also why when reporting the performance of a model variance isn't included? Isn't this also an important factor for assessing model's performance?
I am not speaking about hyperparameters tuning. I suppose that we know the best values for the hyperparameters and we need to estimate the generalization error. My question arised by the fact that we just evaluate once on the test set.