How to aggregate the metrics from two different regression problems?

Question

I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a regression of course). But the way I see it, I cannot simply aggregate the metrics (MSE, RMSE, and MAE) from different problems with each other. Since different problems might be of different ranges (one could be from 0 to 1, and the other from 10 to 100).

But what if I normalize (standardize) each problem's target and also the model's output (with the same average and standard deviation) before calculating the metrics? Does that make the aggregation possible? Is there any other way to make aggregation possible for the metrics of different problems?

Answer 1

Indeed, you cannot aggregate because two problems may predict the price of stocks in dollars and the squared meters of houses, respectively.

Your solution is feasible but cannot adapt continually; the mean and standard deviation may change if new data arrives, and thus, both statistical moments must be recalculated.

Distinctively, it is not strictly necessary to aggregate the metrics if your task is to select the best predictive model among many. In such a case, you can use the Critical Difference Diagram (CDD), as pointed out here.$^\dagger$

---

$^\dagger$_{Please pay attention when interpreting CDD's results: higher metric values are better in classification tasks, while lower ones are preferred in regression.}