CycleGAN: Generator losses don't decrease, discriminators get perfect

Question

So I´m training a CycleGAN for image-to-image transfer.

The problem is: while the discriminator losses decrease, and are very small now, the generator losses don't decrease at all.

The generator loss is:

1 * discriminator-loss + 5 * identity-loss + 10 * forward-cycle-consistency + 10 * backward-cycle-consistency

Somehow the discriminator models, although not being that big, become very strong quite fast, while the generators don't seem to learn much at all.

What do I have to change to resolve this problem and make the generators more effective?

Answer 1

This usually happens when your discriminator starts to just understand that the difference between your generated(fake) and real samples only differ by some n/255 factor. This is easy to learn, and if your generator cannot properly generate "good" samples then within a few 10-100 epochs your Discriminator will learn this highly trivial transformation.

Now when we reach a point where the generator cannot converge from pure random output it will get stuck generating the same random output with a constant high loss. Since I don't know the exact architectures of your network there are some solutions out of the box that has helped me.

Try adding some gaussian noise at the very beginning of your Discriminator network, this will make sure your Discriminator will not learn trivial transformations right away. I cannot comment on this channel yet, so if you can post your network architecture then I could possibly give you a better answer.