3
$\begingroup$

I have implemented a vanilla GAN which gave good results very fast but it had a lot of mode collapse issue, because of this I learned about WGAN which suppose to fix this, in fact they claim they have never encountered mode collapse problem which is great. My main issue is how slow WGAN takes to converge to good results. With a vanilla GAN by epoch 10 I was getting good looking generations, with WGAN I am still seeing noise images on epoch 50! Besides bumping up the learning rate is there anything else I can do? The network is a modified smaller DCGAN that has 1 less output layer than the original. So instead of the extra 1024 output layer mine stops at 512. I knew WGAN was going to be slow from all the research I have done but I didn't except to be this slow. Can some one point me to the right direction on how to better optimize WGAN's speed?

I have googled around but most results are just vague comment chains or agreements that there is a speed problem with no further leads on what you should do next to counter this issue.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

If you are using WGAN with Gradient Penalty, I think the Framework you are using is the limited factor since computing all the gradients will take time.

If you are using WGAN with Gradient Penalty one way to get faster results is to omit the gradient penalty and just do weight clipping as mentioned in original WGAN Paper. But be careful in Improved WGAN (with gradient penalty) they showed, weight clipping can lead to bias in the discriminator.

But for my experience with GANs WGAN GP gave the overall best results and I will spend the time training for it.

But also your training routine could be part of the problem since lot of reference implementations try to get an rate of which the generator updated, e.g. 1 generator update for 10-100 discriminator updates (which in some scenarios totally makes sense). So there are several factors for slow training but most important is that gradient penalty is computational expensive.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Hi, this is an old question and I have already solved it somehow. I think there was some mistake in how I coded the model. I don't remember what I did wrong exactly since it was a while ago but thanks for the effort. I am going to mark this as the correct answer. $\endgroup$
    – Inkplay_
    Commented Jun 1, 2018 at 19:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.