0
$\begingroup$

Two classifiers need to be trained simultaneously, and I have three losses, as shown in the figure. Classifiers 1 and 2 will be updated by losses 1 and 2. Furthermore, loss 3 should update the two classifiers concurrently. Here's what I did

loss1.backward()
loss2.backward()
loss2.backward()

Is this correct? enter image description here

$\endgroup$

1 Answer 1

1
$\begingroup$

You need to create another loss $loss_T$ that combines losses $loss_1$, $loss_2$ and $loss_3$, and optimize only $loss_T$.

The typical approach is to define $loss_T = \lambda_1 \cdot loss_1 + \lambda_2 \cdot loss_2 + \lambda_3 \cdot loss_3$, where $\lambda_1$, $\lambda_2$ and $\lambda_3$ are new hyper parameters (note that you can remove $\lambda_3$ to simplify and still keep the same expressivity). To give values to $\lambda_i$ you may simply assign $\lambda_i = 1$ or you can grid search to obtain better values.

Normally, you want $\lambda_i$ to compensate for the differences in the gradient norms of $loss_i$, to avoid one of them overshadowing the rest. For that, I suggest you monitor the norm of the gradients of $loss_i$ in a training rehearsal to understand the value ranges that are appropriate for each $\lambda_i$.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.