3
$\begingroup$

I'm confused about the expectation notation in the context of GAN loss functions.

The GAN loss for the discriminator is binary cross-entropy. ie: is this real or not.

real = $D(x)$ (ie: give the discriminator a real image)
fake = $D(G(z))$ (ie: generate a fake image and ask discriminator what it is)

Then the binary crossentropy is:
$$log(p) - log (1-p)$$

When used as a GAN loss we replace p with either "real class" or "fake class".

$$log(real) - log (1-fake)=\\ log(D(x)) - log (1-D(G(z)))$$

So far, this is ok (i think haha).

But the actual formulation adds an expectation sign... which I don't understand why it's there.

$$E_{x~data}log(D(x)) - E_z log (1-D(G(z)))$$

$\endgroup$

1 Answer 1

3
$\begingroup$

The functions $$ L_a(x) = \log(D(x))\;\;\;\;\&\;\;\;\; L_b(z) = \log(1-D(G(z))) $$ are defined for one value (either $x$ or $z$).

We don't want to optimize $L_a$ or $L_b$ for one input, we want to minimize it for all possible inputs $x$ and $z$. Suppose $x\in X\equiv\mathbb{R}^n$ (e.g., by "unfolding" it as an image) and $z\in Z\equiv \mathbb{R}^m$. Then we can write minimizing over all possible inputs as minimizing $\int_{\mathbb{R}^n} L_a(x)\,dx $ and $\int_{\mathbb{R}^m} L_b(z)\, dz$. But there's something wrong with this! In fact, we don't care about, say, pure red images or electron microscope micrographs (if $x$ were images), we only care about "real" photographs, i.e., the "natural images". So instead of optimizing over all $X$ and $Z$, we prefer to optimize over the ones that "matter". We do this by placing a probability distribution on $X$ and $Z$, say $P_x$ and $P_z$, such that sampling from these distributions gives us the $x$ and $z$ values that we care about. Low probability is given to unnatural images $x$; high probability is given to e.g. images of dogs and cats. For GANs, we often choose $P_z$ in advance, as a uniform or normal distribution. (Some people prefer to say that the natural images form a manifold in high-dimensional space; this $P_x$ can be thought of as inducing that structure.)

But how to optimize over these "likely" images. The obvious thing to do is optimize the expected value over these densities. In other words, we want to optimize the values of $L_a$ and $L_b$ that we expect to get if we compute them on samples drawn from $P_x$ and $P_z$. This is a weighted average over all of data space, where the weight is given by the probability of seeing that data point: $$ \mathbb{E}_{x\sim P_x}[L_a(x)] = \int_X L_a(x) p_x(x)\, dx \;\;\;\;\&\;\;\;\; \mathbb{E}_{z\sim P_z}[L_b(z)] = \int_Z L_b(z) p_z(z)\, dz $$ where this integral (expectation) is approximated by a sum or average over a minibatch in practice.

In summary, we add the expectations because we care about how we do across all "important" $x$ and $z$ values, not just one.

The explanation in this post might also be of use.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.