I have a 2 part question.
Context I am learning about GANs and writing my own starting from the very simplest example of adversarial learning (1-parameter node), then implementing a very simple 1-dimensional pattern (1010) learning GAN .. and now I am trying to implement an MNIST learning GAN, before I proceed to more realistic photos.
I have some background in machine learning and data mining (masters from a long time ago) and moderately understand how neural networks work.
You can read about my progress on the initial steps here:
I've read lots of the latest blogs, articles, and watched youtube tutorials but still can't get past 2 key problems with implementing MNIST learning GANS.
Q 1. Am I seeing mode collapse?
After some tweaking and iteration I have a GAN which does learn to generate images which look like they might come from the MNIST dataset. Actually they're not digits yet but they are recognisable pen strokes, and certainly not random noise.
You can see a recent iteration of my pytorch code here: github notebook.
When fed random noise, the trained GAN always generates the same, or extremely similar image. Feeding it 1-shot noise (00001000...) also generates a similar image.
Are GANs supposed to generate just one image? Are the different images we see reported from separate training of the GAN? I thought the idea was a single trained GAN could generate many different images from random input noise. Have I misunderstood it?
Q 2. How To Escape Mode-Collapse?
If the answer above is that a train GAN should output many diverse but valid images, then I have mode-collapse.
I've read extensively and tried many approaches to avoiding mode-collapse but none have worked:
- with / without batch normalization
- with / without maxpooling
- with / without dropout
- with / without label softening
- with / without noise added to both input and target labels, including noise that decays over training time
- various widths and depths for the generator, less so the discriminator
- increasing training time (but poor compute power only allows me about 6-10 epochs on the full dataset)
What I've observed or noted is:
- the discriminator width/depth/architecture is tested first to ensure it has the capacity to learn the multi-class MNIST before it is used in the GAN to avoid the case of a discriminator that can't actually learn to discriminate MNIST
- plotting the error as training proceeds (D error, D error on G input) is useful to show that the D error approaches 1/2 as it should, and that D error on G input approaches 1 (well more like 0.8).
- plotting errors also shows stability or collapse of some sort which helps tune the learning rates for example
- I would have thought adding noise to the input or the target labels might kick the GAN out of any local minima which is mode-collapse but I suspect the theory is more complicated than this
- trying different suggestions for the optimiser parameters doesn't help, I have to find my own optimal tuning, and the learning rates are much lower than what others use, eg Adam lr=0.00002 not 0.001 which causes instability
- higher training epochs result in high contrast images which don't look like the MNIST data which has softer edged strokes .. I was hoping higher training epochs would improve diversity of outputs
One area where I can't find much guidance is on the actual architecture of the discriminator and generator:
- Do they have to be matched but opposite? mine aren't - the discriminator is proven to have the learning capacity and that's it. After that the shallower the better to ensure easier back propagation to the generator.
- The use of deconvolution is common in the generator, but some examples online use simple fully connected mapping. Computationally the deconvolutions have fewer learning parameters and intuitively make sense in building an image.
I'd welcome your thoughts and suggestions.