Differentiating vector with different operation on each elements

Question

I have some idea about how backpropagation would work for a loss function like:

loss=summation(predicted-true)^2

Where predicted and true are vectors of the same length and same operation throughout the elements.

Now in object localization problem my neural network's output vector's $0^{th}$ element would denote probability of particular object being in image and rest 4 would tell about bounding box. Now my loss function would be roughly something like this:

loss=CategoricalCrossentropy(pred[0],true[0])+MSE(pred[1:4]-true[1:4])

This loss function worked fine in tensorflow and my NN localized the object as I expected.

My problem is that I am not able to understand how mathematically differentiation would work when different operations are applied on different elements.

Answer 1

Different operations on different elements don't prevent differentiation in any way.

Lets, say we call your above Loss function:

$$\mathcal L=L_1(\mathbf w) + L_2(\mathbf w)$$,

where $\mathbf w$ represents the weights of your model, $L_1$ and $L_2$ are the two loss functions you have defined using the different outputs of your model. The key point is that mathematically, I don't care which output you used to find $L_1$ or $L_2$, just the fact they both somehow depend on the weights of the model.

Thus, if our aim is to differentiate, we simply get (assuming everything in our model is differentiable):

$$\frac{\partial\mathcal L}{\partial\mathbf w} = \frac{\partial L_1}{\partial \mathbf w}+\frac{\partial L_2}{\partial \mathbf w}$$