3
$\begingroup$

What is the derivative of the Loss Function (Absolute Error) with respect to the feature weights that is used to update the weights?

Couldn't find anything specific about it anywhere.

$\endgroup$

2 Answers 2

4
$\begingroup$

The gradient of MAE is not continuous in $y_{pred} = y_{true}$ and therefore there is no defined (bounded, direction independent) derivative at that point.

Elsewhere you have -1, where $y_{pred} > y_{true}$ and +1 where $y_{pred} < y_{true}$

Usually frameworks like TensorFlow, Keras, etc... use an approximate derivative for that point.

$\endgroup$
4
  • $\begingroup$ If I want to compute a simple regression where I will be updating the weights by differentiating the Loss w.r.t the weight vector, how do I implement it? When I did it by hand, I took two different cases on whether y_{pred} > y_{true} and accordingly, beta_{0} = beta_{0} +- learning_rate/(number of samples) and beta_{1} = beta_{1} +- learning_rate*features/(number of samples). I have a bias term and 1 feature. $\endgroup$ Oct 15, 2019 at 14:31
  • $\begingroup$ May help someone who is having the same question: I used a grid search to try different combinations of the two weights. Much slower than gradient descent in MSE. $\endgroup$ Oct 16, 2019 at 4:17
  • $\begingroup$ I'm not sure if I understand you correctly...to do it by hand you would have to differentiate (by approximation) your error for each component. Meaning for a network with 1 Input, 1 Hidden Node and 1 Output: de/dw, de/dI, de/db, de/dy $\endgroup$
    – Philipp
    Oct 16, 2019 at 15:00
  • $\begingroup$ Going by page 360 of Elements of Statistical Learning, the gradient for absolute error loss is $\textrm{sign}[y_i - f(x_i)]$. The sign function is defined at 0, it is 0. So when $y_{pred} = y_{true}$, the gradient would equal 0. $\endgroup$ May 24, 2021 at 16:03
3
$\begingroup$

You can simply approximate $f(x)=|x|$ by $f(x)=\sqrt{x^2+c}$ where $c>0$. You can also utilize subderivative method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.