Suppose, I have a backbone network(convolutional neural network). After this network ends, the output is fed into two neural networks. Both building on the outputs of the feature extractor(CNN). Now if I want to train this complete network from scratch on two different tasks, the weights of the layers after the backbone network can be updated easily, but how should I update the weights of the backbone network. I mean I can compute gradients with respect to two losses, shall I take the mean for the gradients in the backbone or it has to be some weighted sum? if it is the weighted sum then how would the parameters of the weighted sum be updated?