So I´m training a CycleGAN for image-to-image transfer.
The problem is: while the discriminator losses decrease, and are very small now, the generator losses don't decrease at all.
The generator loss is:
1 * discriminator-loss + 5 * identity-loss + 10 * forward-cycle-consistency + 10 * backward-cycle-consistency
Somehow the discriminator models, although not being that big, become very strong quite fast, while the generators don't seem to learn much at all.
What do I have to change to resolve this problem and make the generators more effective?