1
$\begingroup$

I have code that estimates RMSE for k-fold cross-validation and I think it is correct (from book: Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron)

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_squared_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())



So what about MAE? Should I use (with sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(np.sqrt(-scores)).describe())

or this (without sqrt):

scores = cross_val_score(forest_reg, a, b, scoring="neg_mean_absolute_error", cv=10)
print(pd.Series(-scores).describe())

Also for MAE estimation, it should be -scores or scores?

$\endgroup$
3
  • $\begingroup$ For your second question, why do you take -scores for RMSE? My instinct is to do the same for MAE, but I don’t understand why you’re winding up with the wrong sign in the RMSE calculation. $\endgroup$
    – Dave
    Commented May 2, 2020 at 0:56
  • $\begingroup$ @Dave from book "Hands-On Machine Learning with Scikit-Learn, Keras, and TensorFlow, 2nd Edition by Aurélien Géron": Scikit-Learn’s cross-validation features expect a utility function (greater is better) rather than a cost function (lower is better), so the scoring function is actually the opposite of the MSE (i.e., a negative value), which is why the preceding code computes -scores before calculating the square root. So I am pretty sure RMSE should have -scores. I think MAE should have it too, but I just wanted to reconfirm science I am new to Machine learning. $\endgroup$
    – vasili111
    Commented May 2, 2020 at 1:05
  • 1
    $\begingroup$ I didn’t know that about sklearn. I wonder why they did it that way since it’s so common to talk about loss, but with that the case then you should do -scores for MAE, yes. $\endgroup$
    – Dave
    Commented May 2, 2020 at 1:07

1 Answer 1

1
$\begingroup$

It’s an issue of units.

Compare what you’re doing in RMSE and in MAE.

RMSE is a way of getting MSE back to the original units, like how we take the square root of variance to get standard deviation. This makes more physical sense. Sure, we can make sense of square meters, but what about square dollars?

When you do MAE, you don’t have that squaring action to give you squared units. Consequently, while MSE in in square units, MAE is in the original units.

You could take the square root of MAE, but then you’d wind up with measurements with units of $\sqrt{\$}$ or the square root of whatever units you’re using. The result is that your measure of dispersion is not in the original units, which a probably what you want.

I do not see any use for taking the square root of MAE. If you do, please do share. That would be very interesting.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.