3
$\begingroup$

I am using a generative adversarial deep learning model (GAN) with a hybrid loss represented by a linear combination of four losses with three $\lambda$'s, something like:

$total\_loss = loss_1 + \lambda_1\times loss_2 + \lambda_2\times loss_3 + \lambda_3\times loss_4$.

Is there a way to optimize these $\lambda$'s towards the best performance? provided that the computational complexity is not trivial. If my objective is for $loss_4$ to be minimum, should I use a high value for $\lambda_3$ compared to the other ones? But, how would that affect the overall performance, as all the models used in the GAN might play an important role in the final result.

NB. $loss_1$ is based on MSE measure, while the other losses are based on $l_1$ norm.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

When there is an optimization problem involving more than one objective function to be optimized simultaneously, it is called multi-objective optimization.

Gradient descent methods can still be used to minimize the overall objective function. A paper entitled "Gradient-Based Multiobjective Optimization with Uncertainties" goes into greater detail.

In PyTorch, autograd.backward can handle multiple objectives. You need to provide a list of tensors for backpropagation.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thanks for your answer. Indeed, it is called multi-objective optimization, and I have done this previously with simpler problems using Genetic Algorithms. However, to use autograd.backward, these $\lambda$s have to be part of the learning process, by declaring them as Variables, or as the latest version, setting requires_grad to True. Might worth trying it out. Thanks again! $\endgroup$
    – innuendo
    Commented May 14, 2019 at 8:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.