In Goodfellow I, Bengio Y, Courville A. Deep learning. MIT press; 2016 Nov 10. http://thuvien.thanglong.edu.vn:8081/dspace/bitstream/DHTL_123456789/4227/1/10.4-1.pdf

p. 102 (for example), it is said that with Unsupervised Learning, one usually wants to ''learn the entire probability distribution that generated a dataset'', $p(\vec{x})$.

My question is that I would like to have a better interpretation/understanding of the concept, i.e., can we say that $p(\vec{x})$, for a given an example (represented as a vector) $\vec{x}$, in the dataset, means the ''probability to find this example à priori'' in the wild... or something like that? E.g. if it is an image of a cat of breed X, e.g., $\vec{x_i}$ = ''vectorized image of this cat of breed X'' does $p(\vec{x_i})$ means the probability (we estimate given our limited dataset of course) to obtain this image of this cat of breed X, if we draw a sample from this dataset - and even if we want to generalize - the prob to obtain this cat if we sample the TEST SET. ?

There is a similar question here, but the answer is far from answering anything about the question: What does it mean for the training data to be generated by a probability distribution over datasets

  • 2
    $\begingroup$ I think many people have the same question, and I never found a clear answer. $\endgroup$
    – SheppLogan
    Commented Jun 23, 2019 at 21:36

1 Answer 1


It's more of a theoretical distribution, that a concrete one.

The main idea is this:

we consider all data to have an underlying distribution which generates the data. Through the procedure of creating a dataset we effectively sampled some instances from it. Now I like to think of this distribution as a theoretical notion of all possible data of this type that could ever exist.

Let me give you an example:

suppose we have the cats vs dogs dataset. This dataset contains $25000$ images of cats and dogs. Now we can consider the cat images as samples from a larger population. But what would this population include? All cat images on the web? All cat images in existence? or all cat images that could ever conceivably exist? Let's refer to this as this population as $C$. This population follows a certain distribution (not every image is an image of a cat); this distribution essentially tells us what makes a cat, a cat. Now if I were to take a picture of a cat tomorrow (let's call this $c$), I would have effectively taken a sample out of this dataset (i.e. $c \sim C$).

Where does this come into play in Machine Learning?

Well, generative models essentially try to learn this distribution, and they try to do this through its samples (i.e. our dataset). They look at its samples and try to generalize to identify what distribution spawned them. Essentially they try to answer the question what makes $c$ a sample of $C$?

Furthermore, even discriminative models make several assumptions about the data (e.g. that the samples are independent and identically distributed, that the training and test set follow the same underlying distribution)

More formally

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other, and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example. The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{data}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

- I. Goodfellow et al. "The Deep Learning Book" sec 5.2

I'd suggest reading chapter 5 from this book because the authors explain a lot of well known ML concepts (bias, variance, overfitting, underfitting etc.) through the scope of this data-generating distribution.

Edit after comment suggestion:

The question is essentially,

how does the data-generating distribution fit into the training process of a neural network?

The answer isn't so obvious, mainly because neural network classifiers are discriminatve models (i.e. they don't try to identify the data-generating distribution; rather they try to find out what features separate the classes among one another). Also I'd like to add that, as stated previously, the data-generating distribution is a theoretical concept, not a concrete one employed during training.

There is a way, through, we can tie this into the whole training procedure. Initially, consider that NNs try to minimize the cross-entropy loss between its predictions $\hat y$ and the actual labels $y$:

$$ Loss(y, \hat y) = - \sum_i y_i \, log \, \hat y_i $$

Now let's think of $y$ and $\hat y$ not as tensors but as probability distributions. The first represents the probability that a sample would belong to class $y$, while the second represents the probability with which the network thinks a sample belongs to that class.

We can take this one step further and compute the KL divergence between $y$ and $\hat y$. This metric essentially tells us the difference between two distributions (higher values mean distributions are more different).

$$ KL \left( y \|\| \hat y \right) = \sum_i y_i \, log \, \frac{y_i}{\hat y_i} $$

Note that minimizing cross-entropy is equivalent to minimizing the KL divergence between these two distributions. If the two distributions are identical their KL divergence has a value on $0$; this value increases the more they differ.

Minimizing the KL divergence between two distributions is the same as minimizing the JS divergence between them. This is a metric derived from KL, which can be used as a distance function between distributions (i.e. *how close $y$ is to $\hat y$).

So if you think of it this way, Neural Networks are trained to minimize the distance between the actual data-generating distribution $y$ and their perception of the data-generating distribution $\hat y$.

In order for this to be achievable, some assumptions must be held:

  • The samples we have must be representative of the distribution (i.e. $y_i^{train} \sim y$).
  • The test samples we'll use to evaluate our network on must follow the same distribution (i.e. $y_i^{test} \sim y$).
  • The network must have sufficient capacity to learn this distribution.
  • The right optimization strategy needs to be followed to minimize the distance.
  • etc.
  • $\begingroup$ thanks, nice answer but could you continue a bit more on the more "concrete" side? I.e., is/how is it related to the cross-entropy loss? e.g. when you have a vector of probabilities output by your NN and you know your true targets a 1-hot vector with pr=1 for the true class and pr=0 otherwise. $\endgroup$
    – SheppLogan
    Commented Jun 26, 2019 at 19:35
  • $\begingroup$ @Machupicchu I edited my post to best explain how the data-generating distribution relates to a network's training. $\endgroup$
    – Djib2011
    Commented Jun 26, 2019 at 21:43
  • $\begingroup$ Thanks, this is a nice explanation yes. $\endgroup$
    – SheppLogan
    Commented Jun 27, 2019 at 10:28
  • $\begingroup$ but it seems that when minimizing the cross-entropy (X-entropy for short) vs minimizing KL divergence, the difference is that the KL will go to 0 as a minimum and x-entropy to the actual entropy of the sample distribution which if we have a 1-hot vector will be actually 0 and only in that case it will actually be equivalent to minimize X-entropy or KL. Right? $\endgroup$
    – SheppLogan
    Commented Jun 27, 2019 at 10:38
  • 1
    $\begingroup$ Nice thanks, It's always very rewarding to understand something. $\endgroup$
    – SheppLogan
    Commented Jun 27, 2019 at 15:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.