2
$\begingroup$

I am trying to understand Ian Goodfellow et al's paper Generative Adversarial Nets here.

In section 3 the author's write:

The adversarial modeling framework is most straightforward to apply when the models are both multilayer perceptrons. To learn the generator’s distribution $p_g$ over data $x$, we define a prior on input noise variables $p_z(z)$, then represent a mapping to data space as $G(z; $ $\theta_g)$, where $G$ is a differentiable function represented by a multilayer perceptron with parameters $\theta_g$.

I have a few questions.

1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data $x$?

2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Any insights appreciated.

$\endgroup$
2
$\begingroup$

Question 1

1) From what I understand, the generator is a multi-layer perceptron input with a random vector sampled from a predefined latent space, (where wikipedia gives the example of a multivariate normal distribution). What does it mean to learn the generator's distribution $p_g$ over data x?

Generally, supervised learning means to learn a distribution $p_{model}(y\mid x)$ where $x$ are your examples and $y$ your labels. However, in unsupervised learning it is not about labels but instead you are interested in unconditional probabilities and train a model to learn $p_{model}(x)$ (e.g. see p.485/486 in "The Elements of Statistical Learning" by Hastie et al). When applying GANs this $model$ is the generator $g$, i.e. you are learning $p_{g}(x)$. Accordingly, in the original paper the authors state:

The generator $G$ implicitly defines a probability distribution $p_g$ as the distribution of the samples $G(z)$ obtained when $z ∼ p_z$.


Question 2

2) What does "we define a prior on input noise variables $p_z(z)$" mean? I understand that in Bayesian statistical inference, that a prior probability distribution, called a prior for short, gives a probability distribution on outcomes before some evidence is collected.

Prior means we are talking unconditional probabilities, i.e. $p(z)$ is not conditioned on anything related to the task and examples on hand. While for example, $p_d(y\mid x)$ (the distribution learned by the Discriminator) is conditioned on $x$. Often $z$ is sampled from the standard normal distribution $\mathcal{N}(0,1)$.

In "NIPS 2016 Tutorial: Generative Adversarial Networks" the authors provide a good summary of the overall concept of $G$:

The generator is simply a differentiable function $G$. When $z$ is sampled from some simple prior distribution, $G(z)$ yields a sample of $x$ drawn from $p_{model}$. Typically, a deep neural network is used to represent $G$.

$\endgroup$
2
  • $\begingroup$ When the authors write : " To learn the generator’s distribution $p_g(x)$", do they mean for any set of parameters or for a fixed set of parameters? Because this seems if it is the latter, that we are learning a family of probability distributions for each $\theta_g$? $\endgroup$ – IntegrateThis Mar 17 '20 at 22:36
  • $\begingroup$ @IntegrateThis $p_g(x)$ does not depend on $\theta$. Only $G$ does. So I'd say you are learning $p_g(x)$ by training $G(z_g;\theta_g)$ so $G(z_g;\theta_g)$ produces samples drawn from $p_g(x)$. $\endgroup$ – Sammy Mar 18 '20 at 8:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.