MAD vs RMSE vs MAE vs MSLE vs R²: When to use which?

Question

In regression problems, you can use various different metrics to check how well your model is doing:

• Mean Absolute Deviation (MAD): In $[0, \infty)$, the smaller the better
• Root Mean Squared Error (RMSE): In $[0, \infty)$, the smaller the better
• Median Absolute Error (MAE): In $[0, \infty)$, the smaller the better
• Mean Squared Log Error (MSLE): In $[0, \infty)$, the smaller the better
• R², coefficient of determination: In $(-\infty, 1]$ not necessarily the bigger the better

Are there any strong reasons not to use one or the other?

Answer 1

Well actually these can give you different insights into your models errors. If $y$ is your target, $p$ your prediction and $e = p - y$ the errors:

• Mean Error: $ME = mean(e)$
  

In (-∞,∞), the closer to 0 the better.
Measures additive bias in the error. Unbiased estimates should have the same mean as your target thus ME should be close to 0, if it's positive your predictions overestimate the target, if it's negative they underestimate.

• Root Mean Squared Error: $RMSE = \sqrt{mean(e^2)}$.
  

In [0,∞), the smaller the better.
Measures the mean square magnitude of errors. Root square is taken to make the units of the error be the same as the units of the target. This measure gives more weight to large deviations such as outliers, since large differences squared become larger and small (smaller than 1) differences squared become smaller.

• Mean Absolute Error: $MAE = mean(|e|)$.
  

In [0,∞), the smaller the better.
Measures the absolute magnitude of errors and it's units are the same as the units of the target. Makes for more easily interprectable errors and gives less weight to outliers. However a model with good $MAE$ can have punctually very high errors.

• (Root) Mean Squared Log Error: $MSLE= mean((log(p+1)-log(y+1))^2)$.
  

In [0,∞), the smaller the better.
This is useful when dealing with right skewed targets, since taking the log transform makes the target more normally distributed. In practice it's usually achieved by changing the target to $\hat{y}=log(y+1)$ and then predicting as $y=e^\hat{y}-1$

• Median Absolute Deviation: $MAD = median(e - median(e))$.
  

In [0,∞), the smaller the better.
This is a spread metric similar to standard deviation but meant to be more robust to outliers. Instead of taking means of squares as the sd, MAD takes medians of absolutes making it more robust.

• R², coefficient of determination:
  

In (−∞,1] the closer to 1 the better Is a measure of the ratio of variability that your model can capture vs the natural variability in the target variable.

In practice I usually use a combination of $ME$, $R^2$ and: $RMSE$ if there are no outliers in the data, $MAE$ if I have a large dataset and there may be outliers, $RMLSE$ if the target is right skewed.

This link offers a very nice overview on the matter: http://www.cawcr.gov.au/projects/verification/#Methods_for_foreasts_of_continuous_variables

Answer 2

I have very rough ideas for some:

• MAD if a deviation of 2 is "double as bad" than having a deviation of 1.
• RMSE if the value deteriorates more quickly - punishes outliers hard! (can be good or bad)
• MAE if I'm not interested in complete outliers, but only in "typical" cases (as I usually fence the outputs to a reasonable range, this is almost the same as MAD)

For MSLE and R², I have no idea when it is better suited than the others.