0
$\begingroup$

There are several "classical" ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...

These metrics however do not take "costs" into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6).

How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.

$\endgroup$
1
  • 1
    $\begingroup$ Note there is a difference between a cost/objective function which is used to directly drive an optimisation algorithm e.g. by some variant of gradient descent, versus a metric/evaluation function which is used to assess trained models. Which are you looking for? Can you already quantify the difference between under- and over-predicting in your case - e.g. do you have a business cost expressed financially for each type of error? $\endgroup$ Nov 20 '17 at 12:05
1
$\begingroup$

A loss function and cost function are the same thing. As you intuit, classical regression treats loss/cost as symmetric, which is not always what you want. In classification tasks, you can make an asymmetric loss matrix. You can do a similar thing with regression if you solve it with gradient descent, but the ordinary least squares has symmetric loss baked in.

So I would consider either (1) using a numeric optimization library like sklearn or tensorflow to explicitly define the regression parameters you want to estimate, write your own custom loss function, and then do parameter estimation via gradient descent, or (2) finding a software package that allows for asymmetric loss, for example see this discussion.

$\endgroup$
2
  • $\begingroup$ A common idiom I have seen does differentiate between the loss function and the cost function: A loss function measures a single metric, on a single data example, and a cost function consists of an aggregate of loss functions (including both averaging loss functions over many data points, and summing different loss functions). I'm not 100% certain that this is formalised, but it is very common and should probably be mentioned in your answer. $\endgroup$ Nov 22 '17 at 9:54
  • $\begingroup$ That's true. Andrew Ng uses cost/loss in that way in his Coursera courses. I'm not sure the distinction makes a difference for the context of this question, but I certainly take your point. $\endgroup$
    – tom
    Nov 22 '17 at 15:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.