Suppose I have a lot of points $ x_i \in \mathbb{R}^N $ with corresponding non-negative labels $ y_i \in \mathbb{R} $ and I want to do regression and make a prediction on some new datapoint $ x^* \in \mathbb{R}^N $ for which I don't have a label. Is there a name for the procedure of choosing a parametric model $ f_\theta : \mathbb{R}^N \rightarrow \mathbb{R} $ so as to minimize the cost function $ \sum_i {|\log(f_\theta(x_i)) - \log(y_i)|^2 } $ rather than $ \sum_i{|f_\theta(x_i) - y_i|^2} $? It seems that minimizing the difference between logs has some nice properties, and I'm surprised I didn't see this discussed in Bishop's machine learning book, for example.

I thought of this when I was considering a house pricing problem, where I figured I cared more about the percentage by which I was wrong than the pure difference. After all, in my application (and I'm sure many others like it), being wrong by \$50,000 is terrible for a \$60,000 home, but it's okay for a $2.5M home.

Any data science veterans reading this who have used a cost function like the one I suggested above with logs, or who can tell me what it's called (if it has a formal name)?

  • $\begingroup$ Is there any reason why you would rather adjust the loss function instead of just applying a log transformation on your target variable $y$? $\endgroup$ – Sammy Jan 22 at 21:37
  • 1
    $\begingroup$ Nope, that's actually a really good point - probably just better to transform the data with logarithm, train the model, and then use the exponential function in production to get predictions. Doing regression on the logarithm will probably give me better-distributed target values, too. $\endgroup$ – Jake Mirra Jan 22 at 21:41

There is a loss called Root Mean Squared Log Error (RMSLE): $\sqrt[]{\frac{1}{n}\sum_{i=1}^n{(\log(y_i + 1) - \log(\hat{y_i} + 1))^2}}$ (do not forget the $+1$ as the $log$ is not defined at $0$)

You will find a brief explanation and discussion here. It has also been used in competitions as for example here.

| improve this answer | |
  • $\begingroup$ Perfect, upvoted and accepted with my thanks. $\endgroup$ – Jake Mirra Jan 23 at 1:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.