2
$\begingroup$

enter image description here

What is the meaning of $V(D,G)$? How do we get these expectation parts? I was trying to understand it following this article: Understanding Generative Adversarial Networks (D.Seita), but, after many tries, I still can't understand how he got from $\sum_{n=1}^{N} \log D(x)$ to $\mathbb{E}(\log(D(x))$.

$\endgroup$
3
  • $\begingroup$ Welcome to SE.DataScience! this question is more related to probability than GAN. That is, for $x_i \overset{i.i.d.}{\sim} p(x)$ ($x_i$'s independently sampled from distribution $p(x)$), $${\Bbb E}\left[\sum_{n=1}^{N}f(x_n)\right]= \sum_{n=1}^{N}{\Bbb E}[f(x)] = N {\Bbb E}[f(x)]$$ $\endgroup$
    – Esmailian
    Apr 13, 2019 at 13:00
  • $\begingroup$ @Esmailian ok, but why is there expectation instead summation in the first place? Is it same to minimize expectation or to minimize function? I think it doesnt really matter how is this objective function derived, but it is really bothering me not knowing how it is done $\endgroup$
    – i_rezic
    Apr 13, 2019 at 14:51
  • $\begingroup$ Summation is the approximation of that expectation x N. In practice, we minimize on a sum of limited samples rather that the ideal expectation. This is how they are related. For example, average of height of 200 person $(\sum_{n=1}^{200}h_n)/200$ is the approximation of expectation of height in a country ${\Bbb E}[h]$. $\endgroup$
    – Esmailian
    Apr 13, 2019 at 14:58

1 Answer 1

4
$\begingroup$

You are thinking of it backwards :) i.e. don't ask "How do we get these expectation parts?" or "how he got from" sum to expectation, ask where we got the sums!

The expectations are what we actually care about. The sums are just what we use to approximate them.

Generally in ML we define the data space $\mathcal{X}$ (e.g., the set of all possible images) and a probability distribution $\mathcal{P}$ on that space (so that sampling from it gives e.g., the set of natural images). The goal is then to find a function $f$ such that the expected loss $ \mathbb{E}_{x\sim\mathcal{P}}[\mathcal{L}(f(x))] $ is minimal. (This is a weighted average over all possible images, weighted such that only the images we care about are considered). This is the basic tenet of empirical risk minimization, upon which most ML theory is based (often, implicitly). Hence, most papers that are even remotely formal will state their loss functions in terms of expectations or integrals, not sums (since one cannot usually sum over a continuous space in a meaningful way).

In practice, we cannot access $\mathcal{P}$ (indeed, the whole point of GANs is to implicitly model it!). Instead, we have only a dataset $X\subset\mathcal{X}$ sampled as $X\sim \mathcal{P}$. For some loss function $\mathcal{L}$, we can Monte Carlo estimate the real expected $\mathcal{L}$ via $$ \mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right] = \int \mathcal{L}(f(x)) p_\mathcal{P}(x)\, dx \approx \frac{1}{|X|}\sum_{x\in X} \mathcal{L}(f(x)) $$ but even this sum is too hard (expensive) to compute, so we take a subsample (minibatch) $M\sim\mathcal{U}(X)$ uniformly from $X$ instead: $$ \mathbb{E}_{x\sim\mathcal{P}}\left[\mathcal{L}(f(x))\right] \approx \frac{1}{|M|}\sum_{x\in M} \mathcal{L}(f(x)) $$

So for

but why is there expectation instead summation in the first place? Is it same to minimize expectation or to minimize function?

The answer is that we cannot compute the expectation, but we can approximate it with the sum or mean over a minibatch instead, so we use that. Ultimately we define our losses via expectations, and derive the sums to compute from those.

Now for the GAN specific stuff.

What is the meaning of $V(D,G)$?

The point of GANs is learn two functions: $G$, which outputs $x=G(z)$ for $z\sim Q$, and $D$, which outputs a high value for real data ($x\sim P_R$) and low value for fake data ($x\sim P_G$, which we sample via $z\sim Q$ then $x=G(z)$).

What is a good loss function for $D$? Well, we want high values for real data, so let's try to maximize $\log D(x)$ where $x\sim P_R$ is any real data point. So we want the maximum expected value over all of the true dataset, i.e. $\mathbb{E}_{x\sim P_R}\log D(x)$

Next we want low values for fake data, so let's try to maximize the expected $\log(1-D(x_g))$ over all generated fake data $x_g\sim P_G$. But we can equivalently say $x_g=G(z)$, for some $z\sim Q$. So we can instead maximize $\mathbb{E}_{x_g\sim P_G}[\log(1-D(x_g))]=\mathbb{E}_{z\sim Q}[\log(1-D(G(z)))]$.

Putting the two together, we see that we want $D$ to maximize both these terms, so we'd like $$ \max_D \mathbb{E}_{x\sim P_R}[\log D(x)] + \mathbb{E}_{z\sim Q}[\log(1-D(G(z)))] =: \max_D V(G,D)$$ to get a good $D$.

What about $G$? Well, we want $G$ to make $D$ do poorly. But which $D$? The best one. In other words, we want $G$ to make the best $D$ (for that $G$) score as poorly as possible, i.e., $$ \min_G \max_D V(G,D) $$ is our objective. That's where $V(G,D)$ comes from.

Now, how to get the sums from the expectations in $V(G,D)$? It's just a Monte Carlo sample approximation (i.e., a sum or mean over a minibatch) of the expectation, as before. :)

$\endgroup$
1
  • 1
    $\begingroup$ So expectation was the fancy word for summation over the batch. MY GOD, my life has a meaning now! $\endgroup$ Nov 19, 2019 at 3:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.