1
$\begingroup$

I'm not very familiar with neural networks, however, I though I understood the concept of back propagation as starting from the error in the output layer. Say, we have 3 neurons in the output layer and their respective values end up being:

[1 0.5 0.3]

And we wished to obtain values

[0 1 0]

So we can compute a vector of errors between the two:

[-1, +0.5, -0.3] (Not necessarily with the - operation, but you get the point)

And back propagate from there. However, in a the generator of a GAN, it seems to me that the output layer has a bunch of neurons (representative to whatever size the entity we want to generate is) but the error is based only on a % of images the discriminator classified wrongfully. So how exactly do we do back-prop for the generator ?

The only human readable examples of GAN I've found use frameworks and libraries like autoGrad that kind of obfuscate the problem for me :/

Is there a simple way to explain how this backprop can be done ? (Like, in code, not with an "intuitive example" or "simple 4 variable equation where each of the 4 variable packs 50 years of mathematics behind it")

$\endgroup$
1
  • $\begingroup$ A GAN is not a good means by which to learn how backpropagation works at a fundamental level because it obfuscates much of the process by stacking it into a more complex framework. This is like you do not worry about the individual molecular interactions when operating a hammer. You simply package those interactions and approximate their interactions such that we can use them as blocks in our understanding of the problem. Learning backpropagation through a GAN framework over-complicates it needlessly with tons of hurdles that have been solved in simpler contexts in the past. $\endgroup$
    – JahKnows
    May 24 '18 at 2:28
3
$\begingroup$

There is really nothing special about the backpropagation algorithm in a generative adversarial network (GAN). It is the same as that of a convolutional neural network (CNN), as CNNs are usually what the generator and discriminator of the GAN are made of. I will assume the MNIST toy example for the explanation and I will provide code to get a GAN working below.

The GAN

A GAN is composed of a discriminator and a generator. Each of these are held constant while training the other. We will thus alternate between training the discriminator and the generator. This is done separately. Training the discriminator is much easier so lets look at that, then we will look at training the generator as you asked in your question.

enter image description here

Training the discriminator

The discriminator has two output nodes and is used to distinguish real instances from artificial instances. To train the discriminator we will generate $m$ instances using forward passes from the generator, these are the artificial instances, their label will be $y = 0$. To generate these we simply pass a noise vector as the input to the model. We will also use $m$ instances from the real data these will have labels $y = 1$.

enter image description here

We can then see that the discriminator is trained exactly the same way as a basic classification CNN with 2 output nodes. I will describe the training process of a CNN below.

Training the generator

When we train the generator we will keep the discriminator fixed, this is necessary as to not saturate our parameters and make the discriminator too strong to ever beat. So what we essentially have is a CNN (the generator) connected to another CNN (the discriminator). The connecting nodes between these two models is going to be output which will generate the desired images once trained. Do note that the generator wants the instances it will produce in this case to be classified by the discriminator as being from the real distribution, so they will have labels $y = 1$.

enter image description here

All together this is just a CNN and the backpropagation will be computed in the exact same way. First we pass through a noise vector, it goes through the generator, some random image gets produced at its output, that then goes through the discriminator and gets classified as $artificial$. But, we are expecting the discriminator to be fooled in this case, so this is an error, it should have been labeled as $real$. We then use backpropagation to get the error contribution of each model parameter. We will then use gradient descent to update all the parameters associated with the generator.


Backpropagation

This is a method used to compute the contribution of each parameter on the error term. We then use gradient descent to update these parameters such that the next pass through should result in a lower loss rate. Picking the right loss function is essential for this process. For classification tasks, as is the case with a GAN, we typically choose binary cross entropy as defined by

$L = - ylog(\hat{y}) - (1-y)log(1-\hat{y})$

and over $N$ instances as is typically the case for stochastic gradient descent the loss function is

$L(w) = - \frac{1}{N} \sum_{n = 1}^N \big[ y_n log(\hat{y}_n) + (1-y_n)log(1-\hat{y}_n) \big] $

where $y$ is the true label and $\hat{y}$ is the predicted label.

Backpropagation in deep neural networks

Take a look at this answer here which describes the process of using backpropagation and gradient descent to train a single neuron perceptron, and then a multi-layered network.

The only difference is we are using the binary entropy loss function here which has a different derivative with respect to $\hat{y}$. This becomes

$\frac{\partial L}{\partial \hat{y}} = - \frac{1}{N} \sum_{n = 1}^N \Big[ \frac{y}{\hat{y}} - \frac{1-y}{1-\hat{y}} \Big]$

You will then backpropagate this loss through the network by using the chain rule with the activation functions you selected at each layer.

Backpropagation in a CNN

Please refer to this answer for extensive details and a derivation of backpropagation for a CNN. The process is very similar to that for a deep neural network. However, the CNN uses the cross-correlation function at each layer, so you need to backpropagate the loss function through the derivative of this function. That question asks about the inner working of a CNN with two outputs very much like our discriminator.

Finally, in order to train the generator, just imagine the same process with more convolutional layers ahead of it. When we have the associated gradient $\nabla L(w)$ for each parameters, only apply the gradient descent algorithm to the parameters that are a part of the generator model. I am sure that through these derivations you will see that the number of intermediate nodes between layers does not affect the backpropagation algorithm. It remains the same process. You should also be convinced that for a non-human the intermediate nodes that are the generated image from the GAN, are no different from any intermediate nodes in the model. We simply train these intermediate nodes in such a way that we can perceive some meaning, such as generating instances of the MNIST dataset.

$\endgroup$
1
  • $\begingroup$ Often times you can trust past work that have created some technique and just take it at face value, like backpropagation, you can understand it in a fluid way and apply it for use in more complex situations without understanding the nitty-gritty. To truly understand the nuts and bolts of backpropagation you need to go to the root of the problem, and that includes hours upon hours of reading and derivations to convince yourself that it works. It requires linear algebra, advanced calculus, and knowledge of mathematical optimization. I drive my car but I have no idea how the engine works :P. $\endgroup$
    – JahKnows
    May 24 '18 at 4:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.