I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a regression of course). But the way I see it, I cannot simply aggregate the metrics (MSE, RMSE, and MAE) from different problems with each other. Since different problems might be of different ranges (one could be from 0 to 1, and the other from 10 to 100).
But what if I normalize (standardize) each problem's target and also the model's output (with the same average and standard deviation) before calculating the metrics? Does that make the aggregation possible? Is there any other way to make aggregation possible for the metrics of different problems?