2
$\begingroup$

In my GAN, the discriminator loss goes down steadily, while the generator loss oscillates / does not converge.

I suspect this is due to the vanishing gradient problem. Theory: as the discriminator loss doesn't start out big to begin with, the generator will never be backpropagated small gradients into, and therefore it doesn't learn to generate better images.

What are some remedies to resolve this and make the generator actually work?

$\endgroup$
3
  • $\begingroup$ It would very helpful if you could share bits of your code $\endgroup$
    – Leevo
    Jan 31, 2020 at 10:05
  • $\begingroup$ This is one of the bigger problems with GANs, they are usually hard to train. This is an active area of research and it does not seem like this problem has been solved. Without additional information or code, it is hard to help, but usually different loss functions are one way to mitigate the problem. $\endgroup$
    – VincFort
    Jan 31, 2020 at 19:44
  • $\begingroup$ Can you share your architecture and loss function plus a plot of the loss curves? $\endgroup$
    – Jonathan
    Mar 1, 2020 at 20:58

2 Answers 2

0
$\begingroup$

We got it working by simply clipping the gradients on the D network. The code has been pushed into our GAN FHIR gut repo that shows how to do this. The model now learns FHIR and the losses on G and D both decrease together on 200k records.

    # Clip discriminator's gradients
    for p in discriminator.parameters():
        p.data.clamp_(-0.01, 0.01)
$\endgroup$
0
$\begingroup$

A few GAN training tricks, mainly taken from here:

  • Normalize images in [-1,1] use tanh as final layer activation
  • Train the generator on $\max \log(D)$ instead of $\min \log(1-D)$
  • Perform an optimization step of the Discriminator every $n$ Generator steps
  • Avoid sparse gradients: avoid max pooling, use LeakyReLU
  • Use label smoothing 0 → [0,0.3], 1 → [0.7, 1.2] or 0, 0.9 instead of 0,1
  • Use Adam optimizer for Generator and SGD for Discriminator
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.