0
$\begingroup$

I'm about to conduct some tests to compare two solutions to regression problems. And to make the results more robust, I want to apply both on a few different datasets (all problems will be a regression of course). But the way I see it, I cannot simply aggregate the metrics (MSE, RMSE, and MAE) from different problems with each other. Since different problems might be of different ranges (one could be from 0 to 1, and the other from 10 to 100).

But what if I normalize (standardize) each problem's target and also the model's output (with the same average and standard deviation) before calculating the metrics? Does that make the aggregation possible? Is there any other way to make aggregation possible for the metrics of different problems?

$\endgroup$

1 Answer 1

1
$\begingroup$

Indeed, you cannot aggregate because two problems may predict the price of stocks in dollars and the squared meters of houses, respectively.

Your solution is feasible but cannot adapt continually; the mean and standard deviation may change if new data arrives, and thus, both statistical moments must be recalculated.

Distinctively, it is not strictly necessary to aggregate the metrics if your task is to select the best predictive model among many. In such a case, you can use the Critical Difference Diagram (CDD), as pointed out here.$^\dagger$


$^\dagger$Please pay attention when interpreting CDD's results: higher metric values are better in classification tasks, while lower ones are preferred in regression.

$\endgroup$
1
  • 1
    $\begingroup$ Thanks for the answer. I need to study the CDD to understand its use case and see how it applies to my case. But to respond to your statement that I might need to consider the fact that the mean and std dev might change over time, well it does not apply to me. I'm talking about offline datasets which are not changed over time. I'm planning to compare two models in the lab, not the production. Still, thanks a lot for introducing me to CDD. $\endgroup$
    – Mehran
    Jan 28, 2023 at 17:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.