1
$\begingroup$

I am trying to optimize the weights of a Voting Regressor problem. To achieve the best score, I am considering both MAE and MSE as parameters, using the following formula:

score = w * MAE + (w-1) * MSE

Here, I will choose w=0.7 (giving more importance to MAE).

The issue I am facing is due to MSE having a completely different scale compared to MAE. To address this, I plan on using min-max normalization.

  1. Is there another way or metric that takes into consideration both MAE and MSE?

  2. Is min-max normalization the best method, or should I use another type of normalization? If so, why? Z-score won't work since it produces negative values.

$\endgroup$
6
  • 1
    $\begingroup$ Square root of MSE gives a roughly equivalent scale as MAE (same units, at least) $\endgroup$
    – Dave
    Sep 26, 2023 at 9:47
  • $\begingroup$ They are in my case, probably due to the presence of outliers. The RMSE is nearly ten times bigger than MAE. But even if they weren't that appart, the issue still remains. What's the best way to build a score comparing two metrics with different scales? $\endgroup$ Sep 26, 2023 at 19:53
  • $\begingroup$ Well what do you value in the score you calculate? $\endgroup$
    – Dave
    Sep 26, 2023 at 19:56
  • $\begingroup$ Sorry, i didn't understand your question. I was trying to build a scoring method that weights both MAE and MSE using the equation above. In practice i am trying various different voting regressor weights and registering both MAE and MSE. Then, i want to know which voting weights gave me the best combination of scores. I then faced the scale problem and tried to solve it with normalization. $\endgroup$ Sep 26, 2023 at 20:02
  • $\begingroup$ “Best” in what sense? $\endgroup$
    – Dave
    Sep 26, 2023 at 20:03

2 Answers 2

2
$\begingroup$

As you already noticed, MSE (Mean Squared Error) and MAE (Mean Absolute Error) have different scales.

In a regression problem, MSE squares the error, meaning it penalises the model more heavily when the errors are large. On the other hand, MAE measures the the absolute magnitude of the error, meaning it treats small or large errors the same way. Larger errors can usually mean there were outliers in your data, which led to larger errors when using MSE than MAE. So MSE is good in regression problems where minimising large errors is important.

One solution that would not deviate from your original equation would be to use RMSE (Root Mean Squared Error) instead of MSE, as the scale would be the same between them. RMSE is the root of MSE (sqrt(MSE)), hence it is also sensitive to large errors and outliers like MSE. However the difference is that MSE is on the same unit and scale as the values your regression model is outputting, like MAE.

Therefore, in your formula you could replace MSE with RMSE, which will keep both metrics be on the same scale and unit, and you could use w to decide how your score favours a type of model over another:

score = w * MAE + (w-1) * RMSE

For:

  • a regressor with a balanced view, use a higher value of w
  • a regressor with less large errors, use a smaller value of w.

You could fine-tune the value of w to see which regressor best fits your purpose.

$\endgroup$
1
  • $\begingroup$ Hello @Adam, thank you for your answer. Although RMSE and MAE share the same unit, they do not have the same scale in my case. For instance, in my problem, RMSE is approximately 10 times larger (which, as you mentioned, indicates that I have outliers). How would you address this? Is min-max normalization the best way to scale them equally? $\endgroup$ Sep 27, 2023 at 17:45
1
$\begingroup$

For regression problems, it makes sense to use the RMSE or root mean squared error, which is the square root of MSE. This is simply because it is the same units as your data, and therefore easy to quantify the variation from the actual values.

For standardization, I wouldn't say the min-max method is the best because there are other standardization methods that work well depending on your problem. Read through them here to find out which works best for you.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.