I have some idea about how backpropagation would work for a loss function like:


Where predicted and true are vectors of the same length and same operation throughout the elements.

Now in object localization problem my neural network's output vector's $0^{th}$ element would denote probability of particular object being in image and rest 4 would tell about bounding box. Now my loss function would be roughly something like this:


This loss function worked fine in tensorflow and my NN localized the object as I expected.

My problem is that I am not able to understand how mathematically differentiation would work when different operations are applied on different elements.


1 Answer 1


Different operations on different elements don't prevent differentiation in any way.

Lets, say we call your above Loss function:

$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,

where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that mathematically, I don't care which output you used to find $L_1$ or $L_2$, just the fact they both somehow depend on the weights of the model.

Thus, if our aim is to differentiate, we simply get (assuming everything in our model is differentiable):

$$\frac{\partial\mathcal L}{\partial\mathbf w} = \frac{\partial L_1}{\partial \mathbf w}+\frac{\partial L_2}{\partial \mathbf w}$$

  • $\begingroup$ each elements are depended only row of weights. For example predicted[0] is depended on 0th row of weights. So isn't differentiating w.r.t whole w unnecessary? $\endgroup$
    – BRUCE
    Apr 16, 2021 at 2:15
  • $\begingroup$ @BRUCE If a variable is independent of some of the weights (which btw I don't think will be the case in normal neural net architectures), then there is no need to differentiate wrt to those weights... typically you don't have to worry about that if using automatic differentiation in packages such as pytorch or tensorflow... $\endgroup$
    – VRaina
    Apr 18, 2021 at 15:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.