There are several "classical" ways to quantify the quality of (any!) regression models such as the RMSE, MSE, explained variance, r2, etc...
These metrics however do not take "costs" into account, for example, for me it is worse to under-predict a value (Real: 0.5, Predicted: 0.4) than to over-predict it (Real: 0.5, Predicted: 0.6).
How can I model such costs into an evaluation function? I just need a first idea to start with and will welcome any suggestions.