0
$\begingroup$

I was recently trying to explain to someone whether performance of my estimation approach is good or bad. For instance, whether a model with Mean Absolute Error (MAE) of 17000 is a bad solution. It was also hard for me to explain whether performance loss by 225 (in terms of MAE), when switching from one model to another, is significant or not.

To me it was clear that both are little because I knew the context: we we're talking about predicting house prices ranging from \$34,900 (min) to \$755,000 (max), so that

  1. MAE=17,000 is just 2.5% of the difference between max and min
  2. Change in MAE by \$225 is just 0.03% of the difference between max and min.

Are there some normalized metrics for comparing performance of regression models without the need to know the context?

Which of those metrics are available in scikit-learn? For instance, it provides mean_absolute_error and mean_squared_error but they are not normalized.

Edit:

After Joe B suggestions, I've plotted a graph to see deviations between predicted and expected price. In fact, that's gives more insight than a single-number metric:

Plot

$\endgroup$
  • $\begingroup$ Could you elaborate on what you mean by whether or not switching from one model to another might be "significant"? $\endgroup$ – Upper_Case May 24 '19 at 18:13
  • $\begingroup$ @Upper_Case By switching from one model to another I mean for instance: 1) switching from decision tree and random forest or 2) changing data preprocessing approach: switching feature drop to imputing with mean value. Defining what significant mean is actually core part of this question. If you don't know the context, knowing that MAE has dropped by $225 won't tell you much. It would, however, tell you more, if it was normalized, i.e, max possible MAX was 1.0, minimal -- 0.0. Then seeing drop by 0.1 would be significant to me. $\endgroup$ – dzieciou May 27 '19 at 15:18
2
$\begingroup$

A couple suggestions

  1. MAE represents your mean error. This is essential to recognize as having an error of even half of your MAE on the low end of your spectrum ($34,900) is huge. However, to your point, at the high end of your spectrum it is quite small.

  2. To solve the above, you should plot a graph to identify the patterns in your error. You'll plot your predicted y value vs. your actual y value and see where your predictions deviate from the y=x line.

Normalized metrics:

you may use an r-squared value. This might be ideal because its looking at the percent of variance explained by your regression and thus is a relative measure of fit. Your r-squared value will always be between 0 and 1 (assuming your model is not worse than always guessing your mean value) where 1 is perfect. Implement as follows:

from sklearn.metrics import r2_score
y_pred = model.predict(X_train)
print(r2_score(y_true, y_pred))
|improve this answer|||||
$\endgroup$
  • 2
    $\begingroup$ Good answer, I've little to add except: the Akaike Information Criterion may also be a good option for the OP, and none of these evaluations can tell the OP whether or not their model is good or even good enough for some purpose. They can only suggest which (similar) models might be better than one another. $\endgroup$ – Upper_Case May 24 '19 at 18:15
  • $\begingroup$ @Upper_Case You're right that no metric will tell me when to stop in pursing a better model. It really depends on a particular case. If I want to sell a house and estimate the selling price for it, comparing to what's on the market, the prediction error I could accept will depend on how big financial margin I have. Perhaps, for the real estate of \$20,000 the error of $1000 means much more than for the real estate of \$300,000. Now I even see the value of an absolute metric instead of normalized. $\endgroup$ – dzieciou May 28 '19 at 8:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.