Bias-variance trade-off and model evaluation

Question

Suppose that we have train a model (as defined by its hyperparameters) and we evaluated it on a test set using some performance metric (say $R^2$). If we now train the same model (as defined by its hyperparameters) on a different training data we will get (probably) a different value for $R^2$.

If $R^2$ depends on the training set, then we will obtain a normal distribution around a mean value for $R^2$. Shouldn't therefore average the $R^2$ from the various evaluations in order to get a better picture of the models performance? Also why when reporting the performance of a model variance isn't included? Isn't this also an important factor for assessing model's performance?

I am not speaking about hyperparameters tuning. I suppose that we know the best values for the hyperparameters and we need to estimate the generalization error. My question arised by the fact that we just evaluate once on the test set.

Answer 1

Estimating the variance in generalization error is useful and is best assessed through cross-validation (not on train/test split). The data should be split into folds and each fold should be trained with the same algorithm and hyperparameters. Then each training fold should evaluation on its respective validation fold. Given the repeated nature, it is possible to estimate the "spread" of generalization error.

Additionally, $R^2$ is often considered not an appropriate metric to evaluate generalization error because $R^2$ relies on the mean of the training data.