0
$\begingroup$

So here is my network architecture as shown in below image. I’ve two separate networks and loss_1 and loss_2 are coming from two separate nn.Modules networks and final_loss = loss_1 + loss_2.enter image description here

Still only one final_loss.backward() would calculate grad in both networks? I do have two separate optimizers to update grads in two separate networks respectively.

If not so, then what is the correct way to do back propagation in these individual networks on combined loss value.

$\endgroup$
1
$\begingroup$

A single final_loss.backward() would calculate the gradients but there are some strange things in your scheme (these may seem strange due to lack of information in your question):

The first strange this is: why having two separate optimizers? It could only be justified if you purposely want different optimization algorithms for each network.

The second and most strange thing is: why would you want to have a combined loss? The only reason that comes to my mind would be that module1 and module2 share some parameters. Otherwise, I see no point in combining two totally unrelated losses because they could have different scales, leading to one of the partial losses to have little effect. Minimizing each loss separately would lead to a much better result.

$\endgroup$
1
  • $\begingroup$ Thanks @ncasas for the answer. Yes you are right, two optimizers work differently on different parameters. Also, I'm using combined loss because some of the layers are shared. $\endgroup$ – Abhinav Ravi Jul 12 '19 at 5:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.