I was reading this nice tutorial about Pytorch's basics:


In the first example (pure Numpy), the author starts the backward phase by setting as his "first" gradient the numerical result of the loss function calculation multiplied by the constant 2:

grad_y_pred = 2.0 * (y_pred - y)

Why does he multiply by 2?


1 Answer 1


This is because of the loss assumption is the (Mean) Squared Error $\mathcal{L} = (\hat{y} - y)^2$ and the derivative is

$$ \frac{\partial}{\partial \hat{y}} \mathcal{L} = 2 (\hat{y} - y) $$

which is then passed "backward" for use in the chain rule.

  • $\begingroup$ Oh my goodness.. Don't know why I 'removed' the square from my train of thoughts... Thanks, and apologies.. $\endgroup$
    – MadHatter
    Apr 8, 2020 at 11:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.