# How can both generator and discriminator losses decrease?

In this paper there is a plot of how the loss of gans looks through epochs.

Figure 2: These are of course averaged losses.

How can both the discriminator loss and generator loss decrease?

real_y = discriminator(real_sample)
fake_y = discriminator(generator(noise))
discriminator_loss = real_y-fake_y+1 # nicer, it is between 0 and 2

fake_y = discriminator(generator(noise))
generator_loss = fake_y


I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. One optimizer is making fake_y less and the other optimizer is making it more.

Maybe the loss functions aren't calculated like I said.

In the widely used analogy:

In simple terms the generator is like a forger trying to produce some counterfeit material, and the discriminator is like the police trying to detect the forged items.

We can measure how good the police is by how many times out of 100 fakes he can identify the real and fake ones and the forger as deceiving the policeman out of 100 fakes.

Wouldn't that mean that if the police get better, the forger gets worse? (if using the aforementioned measure of how good they are)

Therefore we wouldn't be able to see both of them being good at the same time! But the graph from the paper indicates otherwise.

What am I missing?

• Maybe both discriminator and generator are overfitting, able to fit new batch well. May 14 '20 at 2:40

In the subsection of the figure, they compute the average loss across 100 iterations, which is why the loss is monotonically decreasing because on average the loss does decrease with the training.

You are correct in inferring that if this was reported on an iteration to iteration basis, the loss would be a zig zag curve, which is less nice to look at than a smooth curve.

As in the comments, the loss can monotonically decrease to 0, if it is the Wasserstein distance. The Wasserstein distance is computed as $$\sup_{f \in L_{k}} E_{x \sim P_X}[f] - E_{x \sim P_{\theta}} [f]$$ where the first term is the expectation of the function, $$f$$ (discriminator) over the batch and the second term is the generated data from the generator computed on the discriminator. Note that by this formulation, it returns the $$k$$-Wasserstein distance.

• Good answer - the devil is in the details, but still an interesting observation on a batch-by-batch basis from the OP. Jun 5 '18 at 22:03
• Certainly! @n1k31t4, I find the concept of GAN's to be very fascinating, although I do not know much about it
– PSub
Jun 5 '18 at 22:10
• I realize that it would be less smooth but I don't understand how the two loss functions can both decrease (even when averaged). I would expect one of the losses to increase as the other one decreases. Since they use the same calculation of fake_y and one decreases -fake_y and the other fake_y. Jun 6 '18 at 16:02
• The goal of the training is to find the minimum loss function, so the optimizer optimizes both the discriminator and the generator simultaneously.
– PSub
Jun 6 '18 at 16:16
• How are people up voting this answer? It has nothing to do with the question. Mar 13 at 2:04

This is because the discriminator in WGan isn't a classifier and works differently from the traditional GAN. You are 100% correct that in a normal GAN it wouldn't make any sense for both losses to decrease as shown.

• This isn't true -- despite the WGAN having a different loss function, namely the Wasserstein distance, one should still not expect that the discriminator and generator simultaneously monotonically increase -- generally one of them "wins" the round and receives a lower portion of the loss.
– PSub
Mar 13 at 6:07
• @PSub You are completely misunderstanding the question. It's not a question about the small scale changes of the loss values. OP is asking how they can both move towards zero when one is defined as the inverse of the other. It's because in a WGan the losses are not defined this way. This is one of the useful qualities of the W distance cited in the paper - the loss actually means something about how well the network is converging. In a normal GAN this is not the case. Mar 17 at 6:17
• There are a couple of things that are wrong here. You're right that the Wasserstein metric is indicative of the distance between two distributions. But I think you misunderstand that GANs cannot have a monotonically decrease the loss -- if the generator improves faster than the discriminator deteriorates, you would have a reducing loss. Maybe, we're disagreeing about semantics here, but I'm not sure -- if you look carefully at the figure, the GAN loss also decreases -- are you saying this is incorrect?
– PSub
Mar 18 at 1:45
• Even in the figure G and D are roughly inversely proportionate. There is a lot of noise because at each point in the graph you are measuring their loss for a small sample, but in general as one goes up you see the other go down. As I said one of the nice things about W loss is it doesn't behave this way. That's one of the main things this graph is showing. The losses are interpretable unlike in a normal Gan. Mar 18 at 2:47
• I don't think we're disagreeing about the Wasserstein metric having the capability of decrease to 0. I think what we're disagreeing about is whether the GAN loss can decrease at all. From your comments above it seems like you think that it cannot which is patently false. I agree that it decreasing to 0 is indeed confusing, but I would need to read their code before concluding it's an error.
– PSub
Mar 18 at 3:04