3
$\begingroup$

What is the derivative of the Loss Function (Absolute Error) with respect to the feature weights that is used to update the weights?

Couldn't find anything specific about it anywhere.

$\endgroup$
4
$\begingroup$

The gradient of MAE is not continuous in $y_{pred} = y_{true}$ and therefore there is no defined (bounded, direction independent) derivative at that point.

Elsewhere you have -1, where $y_{pred} > y_{true}$ and +1 where $y_{pred} < y_{true}$

Usually frameworks like TensorFlow, Keras, etc... use an approximate derivative for that point.

$\endgroup$
4
  • $\begingroup$ If I want to compute a simple regression where I will be updating the weights by differentiating the Loss w.r.t the weight vector, how do I implement it? When I did it by hand, I took two different cases on whether y_{pred} > y_{true} and accordingly, beta_{0} = beta_{0} +- learning_rate/(number of samples) and beta_{1} = beta_{1} +- learning_rate*features/(number of samples). I have a bias term and 1 feature. $\endgroup$ – user3656142 Oct 15 '19 at 14:31
  • $\begingroup$ May help someone who is having the same question: I used a grid search to try different combinations of the two weights. Much slower than gradient descent in MSE. $\endgroup$ – user3656142 Oct 16 '19 at 4:17
  • $\begingroup$ I'm not sure if I understand you correctly...to do it by hand you would have to differentiate (by approximation) your error for each component. Meaning for a network with 1 Input, 1 Hidden Node and 1 Output: de/dw, de/dI, de/db, de/dy $\endgroup$ – Philipp Oct 16 '19 at 15:00
  • $\begingroup$ Going by page 360 of Elements of Statistical Learning, the gradient for absolute error loss is $\textrm{sign}[y_i - f(x_i)]$. The sign function is defined at 0, it is 0. So when $y_{pred} = y_{true}$, the gradient would equal 0. $\endgroup$ – Marjolein Fokkema May 24 at 16:03
3
$\begingroup$

You can simply approximate $f(x)=|x|$ by $f(x)=\sqrt{x^2+c}$ where $c>0$. You can also utilize subderivative method.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.