Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
There are several "classical" ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...
These metrics however do not take "costs" into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6).
How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.
A loss function and cost function are the same thing. As you intuit, classical regression treats loss/cost as symmetric, which is not always what you want. In classification tasks, you can make an asymmetric loss matrix. You can do a similar thing with regression if you solve it with gradient descent, but the ordinary least squares has symmetric loss baked in.
So I would consider either (1) using a numeric optimization library like sklearn or tensorflow to explicitly define the regression parameters you want to estimate, write your own custom loss function, and then do parameter estimation via gradient descent, or (2) finding a software package that allows for asymmetric loss, for example see this discussion.
