1
$\begingroup$

In GAN architecture, during training, what keeps the generator's output dependant on the input noise? Why don't the weights of the noise input become zero (plus a bias)? I would expect the generator to converge to outputting a single picture which is extremely real and non-distinguishable from a real picture, and ignore the noise all together, since this is a "cheaper" way (in convergence time and in number of parameters used) to decrease the generator's loss.

$\endgroup$
1
$\begingroup$

If the generator always outputs the same image, then it's easy for the discriminator to win the game and tell apart the output of the generator from random images in the training set: if the input to the discriminator is that one image, then it outputs "came from the generator", otherwise outputs "came from the training step". The game is set up so that the generator is rewarded for fooling the discriminator. Always outputting the same image isn't going to fool the discriminator.

|improve this answer|||||
$\endgroup$
0
$\begingroup$

More concretely, say for instance, you train a GAN on MNIST to teach it to draw realistic digits.

Say, the generator learns to draw a perfect 9 and always draws it independently of noise. Then, the discriminator will quickly learn to discard all 9s, genuine or fake: better be off on 1/10 of the real set (1/20 of the full set) resulting in 5% total error, than guessing (50% proba) over the fake nines (the entire fake set, 1/2 of the full set) resulting in 25% total error.

So, the generator will have to learn to draw something else, perhaps it will try all 5s or something, but it will have to snap out of this pattern: only by matching the distribution of the real set (generate 1/10 of each digit) will it prevent the discriminator to quickly and effectively discard all its productions.

I had a (very) simple implementation, and, effectively, it started by always drawing the same digit independently of the noise, and then, it quickly snapped out of this pattern and started drawing multiple digits. In the end, it drew approx 1/10 of each :)

|improve this answer|||||
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.