3
$\begingroup$

In this paper there is a plot of how the loss of gans looks through epochs.

Figure 2:

figure 2

These are of course averaged losses.

How can both the discriminator loss and generator loss decrease?

real_y = discriminator(real_sample)
fake_y = discriminator(generator(noise))
discriminator_loss = real_y-fake_y+1 # nicer, it is between 0 and 2

fake_y = discriminator(generator(noise))
generator_loss = fake_y

I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. One optimizer is making fake_y less and the other optimizer is making it more.

Maybe the loss functions aren't calculated like I said.

In the widely used analogy:

In simple terms the generator is like a forger trying to produce some counterfeit material, and the discriminator is like the police trying to detect the forged items.

We can measure how good the police is by how many times out of 100 fakes he can identify the real and fake ones and the forger as deceiving the policeman out of 100 fakes.

Wouldn't that mean that if the police get better, the forger gets worse? (if using the aforementioned measure of how good they are)

Therefore we wouldn't be able to see both of them being good at the same time! But the graph from the paper indicates otherwise.

What am I missing?

$\endgroup$
3
$\begingroup$

In the subsection of the figure, they compute the average loss across 100 iterations, which is why the loss is monotonically decreasing because on average the loss does decrease with the training.

You are correct in inferring that if this was reported on an iteration to iteration basis, the loss would be a zig zag curve, which is less nice to look at than a smooth curve.

$\endgroup$
  • $\begingroup$ Good answer - the devil is in the details, but still an interesting observation on a batch-by-batch basis from the OP. $\endgroup$ – n1k31t4 Jun 5 '18 at 22:03
  • $\begingroup$ Certainly! @n1k31t4, I find the concept of GAN's to be very fascinating, although I do not know much about it $\endgroup$ – PSub Jun 5 '18 at 22:10
  • $\begingroup$ I realize that it would be less smooth but I don't understand how the two loss functions can both decrease (even when averaged). I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. $\endgroup$ – Hadus Jun 6 '18 at 16:02
  • $\begingroup$ The goal of the training is to find the minimum loss function, so the optimizer optimizes both the discriminator and the generator simultaneously. $\endgroup$ – PSub Jun 6 '18 at 16:16

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.