# differentiating vector with different operation on each elements

I have some idea how backpropagation would work for loss function like:

loss=summation(predicted-true)^2

Where predicted and true are vector of same length and same operation throughout the elements.

Now in object localization problem my neural network's output vector's 0th element would denote probability of particular object being in image and rest 4 would tell about bounding box. Now my loss function would be roughly something like this:

loss=CategoricalCrossentropy(pred,true)+MSE(pred[1:4]-true[1:4])

This loss function worked fine in tensorflow and my NN localized the object as I expected.

My problem is that I am not able to understand how mathematically differentiation would work when different operations are applied on different elements.

Different operations on different elements don't prevent differentiation in any way.

Lets, say we call your above Loss function:

$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,

where $$\mathbf w$$ represents the weights of your model, $$L_1$$ and $$L_2$$ are the two loss functions you have defined using the different outputs of your model. The key point is that mathematically, I don't care which output you used to find $$L_1$$ or $$L_2$$, just the fact they both somehow depend on the weights of the model.

Thus, if our aim is to differentiate, we simply get (assuming everything in our model is differentiable):

$$\frac{\partial\mathcal L}{\partial\mathbf w} = \frac{\partial L_1}{\partial \mathbf w}+\frac{\partial L_2}{\partial \mathbf w}$$

• each elements are depended only row of weights. For example predicted is depended on 0th row of weights. So isn't differentiating w.r.t whole w unnecessary? – BRUCE Apr 16 at 2:15
• @BRUCE If a variable is independent of some of the weights (which btw I don't think will be the case in normal neural net architectures), then there is no need to differentiate wrt to those weights... typically you don't have to worry about that if using automatic differentiation in packages such as pytorch or tensorflow... – VRaina Apr 18 at 15:19