2
$\begingroup$

I have a 2 part question.


Context I am learning about GANs and writing my own starting from the very simplest example of adversarial learning (1-parameter node), then implementing a very simple 1-dimensional pattern (1010) learning GAN .. and now I am trying to implement an MNIST learning GAN, before I proceed to more realistic photos.

I have some background in machine learning and data mining (masters from a long time ago) and moderately understand how neural networks work.

You can read about my progress on the initial steps here:

I've read lots of the latest blogs, articles, and watched youtube tutorials but still can't get past 2 key problems with implementing MNIST learning GANS.


Q 1. Am I seeing mode collapse?

After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.

You can see a recent iteration of my pytorch code here: github notebook.

When fed random noise, the trained GAN always generates the same, or extremely similar image. Feeding it 1-shot noise (00001000...) also generates a similar image.

Are GANs supposed to generate just one image? Are the different images we see reported from separate training of the GAN? I thought the idea was a single trained GAN could generate many different images from random input noise. Have I misunderstood it?


Q 2. How To Escape Mode-Collapse?

If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.

I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:

  • with / without batch normalization
  • with / without maxpooling
  • with / without dropout
  • with / without label softening
  • with / without noise added to both input and target labels, including noise that decays over training time
  • various widths and depths for the generator, less so the discriminator
  • increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)

What I've observed or noted is:

  • the discriminator width/depth/architecture is tested first to ensure it has the capacity to learn the multi-class MNIST before it is used in the GAN to avoid the case of a discriminator that can't actually learn to discriminate MNIST
  • plotting the error as training proceeds (D error, D error on G input) is useful to show that the D error approaches 1/2 as it should, and that D error on G input approaches 1 (well more like 0.8).
  • plotting errors also shows stability or collapse of some sort which helps tune the learning rates for example
  • I would have thought adding noise to the input or the target labels might kick the GAN out of any local minima which is mode-collapse but I suspect the theory is more complicated than this
  • trying different suggestions for the optimiser parameters doesn't help, I have to find my own optimal tuning, and the learning rates are much lower than what others use, eg Adam lr=0.00002 not 0.001 which causes instability
  • higher training epochs result in high contrast images which don't look like the MNIST data which has softer edged strokes .. I was hoping higher training epochs would improve diversity of outputs

One area where I can't find much guidance is on the actual architecture of the discriminator and generator:

  • Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
  • The use of deconvolution is common in the generator, but some examples online use simple fully connected mapping. Computationally the deconvolutions have fewer learning parameters and intuitively make sense in building an image.

I'd welcome your thoughts and suggestions.

Example output which I think looks good, but is mode-collapsed: enter image description here


$\endgroup$
0
$\begingroup$

Disclaimer:

I am also fairly new to GANs, but I've been extensively playing with things, and trying various ideas to get something usable (I'm also using PyTorch). So I am by no means an expert, but after seeing your question, I thought I'd share some things I've learned along the way with the hope that you'd find them useful. I haven't looked thoroughly at your code yet, so I assume your code is generally correct (meaning no unwanted silly mistakes with the network models, loss computation, etc...).

Also, beware that I'm not using MNIST and my architecture has recurrent layers. So not all these suggestions might be applicable to you and YMMV...


Q 1. Am I seeing mode collapse?

After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.

Remember that mode collapse happens when your network fails to generate a diverse enough set of outputs (most/all samples look the same).

Looking at your example image, I don't think you're even at the point of mode collapse yet. Your example image does not look much like a "real" digit. In my experience, when mode collapse happens, your generator would produce valid and almost convincing looking example, but that's it. All generated examples will look pretty much the same.

That being said, I think your problem is that your generator network has not yet learned to produce real (or semi real) looking samples. I'd suggest training for longer, while making sure that your loss computation, etc. are all correct. When my network started working, I could immediately tell that it's producing valid output.

A few pointers:

  • Be sure that your learning rate is small enough. My first problem was a large learning rate (I used 0.001 with Adam, then realized that my model only works with something small like 0.0002).
  • Make sure learning is happening. Track the loss values over time and make sure they make sense. There shouldn't be any spikes in loss values, else something is wrong.

I've had great success with feature matching as my loss metric. Without feature matching, my network never really worked well.


Q 2. How To Escape Mode-Collapse?

That's the million dollar question! I've been pulling my hair out trying to find a working solution for mode collapse for the past month.

You see, my problem right now is that all generated samples look super convincing, but they all look almost exactly the same.

If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.

I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:

with / without batch normalisation with / without maxpooling with / without dropout with / without label softening with / without noise added to both input and target labels, including noise that decays over training time various widths and depths for the generator, less so the discriminator increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)

It's great that you tried all this (I did too), but I found that for me, no architectural change really made any difference. The only thing that has partially worked for me so far is using the WGAN loss. It's very tricky to get it to work, but after about 2000 epochs, I see samples that look real (with some obvious flaws) but look very different from one another.

I suggest you try both WGAN and WGAN-GP as your loss metric. Due to reasons beyond this discussion, I cannot easily use WGAN-GP.


Lastly:

Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.

My discriminator and generator almost match and are opposite of each other. I've tried many many variations, but didn't find any significant differences/improvements. I just decided to keep things simple and match them for now. Once my mode collapse issue goes away, I will revisit this.

Hope this helps. I really hope to get feedback from some experts on this.

$\endgroup$
  • $\begingroup$ thanks M2X - really helpful suggestions. $\endgroup$ – MYO Algorithmic Art May 14 '19 at 22:25
  • $\begingroup$ do you have any idea about what could be wrong if you see spikes in loss values as you said? $\endgroup$ – deltaskelta Nov 6 '19 at 1:58
0
$\begingroup$

Since this question I have made significant progress, which I have written up as part 3 of the series:

http://makeyourownalgorithmicart.blogspot.com/2019/05/generative-adversarial-networks-part-iii.html

Although my GAN doesn't now have mode collapse - the reasons are still unclear. The post reports visually the impact of combinations of SGD->Adam, none->LayerNorm, Sigmoid->ReLU which I hope others find useful:

gan optimisation

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.