0
$\begingroup$

In the TensorFlow example (https://www.tensorflow.org/tutorials/generative/dcgan#the_discriminator) the discriminator has a single output neuron (assume batch_size=1).

Then over in the training loop the generator's BinaryCrossentropy loss is calculated using the discriminator's output which has the shape [1]. It then calculates the loss gradients by plugging in the prediction and label into the dBinaryCrossentropy derivative whose resultant shape is also [1]. How is this [1] shaped gradient fed backward into the generator's layers when its shape doesn't match and the Conv2DTranspose layer expects gradients whose shape matches its output?

$\frac{dL}{dZ} = \frac{dL}{dA}*\frac{dA}{dZ}$ <--- the first term's shape is [1] but the second term's shape is the same as the Conv2DTranspose output shape, can't do hadamard product. How does backpropagation still work?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

What needs to match is the input of the discriminator with the output of the generator.

The gradient is backpropagated from the [1]-shaped loss back up through the whole discriminator and then through the generator, from its output back to its input.

Improve this answer
$\endgroup$
4
  • $\begingroup$ In this sense do the generator and discriminator form a single nework? The last layer of the generator receives gradients from the first layer of the discriminator? From the paper arxiv.org/pdf/1406.2661.pdf section 4 algorithm 1 can you point exactly where your answer occurs? When it updates the generator by descending its stochastic gradient, ∇log(1 - D(G(z))), D(G(z)) is shaped as [1] which in the TF example is not the same as the generator output 28x28x1. $\endgroup$
    – rkuang25
    Jan 29 at 1:34
  • $\begingroup$ Yes, that's it, they form a single network (except when the discriminator is trained to recognize real images). My answer in grounded in the original GAN paper, which you linked, section 4 algorithm 1, at the parts that use $D(G(x))$ in the update of both discriminator and generator. As you can see, the discriminator receives the output of the generator. $\endgroup$
    – noe
    Jan 29 at 9:36
  • $\begingroup$ Can you confirm that my translation of algorithm 1 form the paper is correct in terms of the TF example? Run forward propagation on generator with noise to get a fake image to run it through discriminator, then run a real image through the discriminator. Calculate the combined loss gradients for these two and do backpropagation on discriminator only. Then run forward propagation for generator with noise again, run that generated image through discriminator, then calculate loss gradients. This is the [1] shaped gradient which is backpropagated through the generator + discriminator network. $\endgroup$
    – rkuang25
    Jan 30 at 8:05
  • $\begingroup$ Your description sounds Ok. You can also take a look at the multiple open-source implementations of GANs on Github. I suggest you open a new question to further discuss this if you find it necessary. $\endgroup$
    – noe
    Jan 30 at 8:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.