What does it mean for the training data to be generated by a probability distribution over datasets

Question

I was reading the Deep Learning book and came across the following para (page 109, second para):

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{\text{data}}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Can somebody please explain to me the meaning of this paragraph?

On page 122 the last paragraph, it also gives an example

a set of samples $\{x(1), \dots, x(m) \}$ that are independently and identically distributed according to a Bernoulli distribution with mean $\theta$.

What does this mean?

Answer 1

There is a common assumption that data that is being modeled is independent and identically distributed (i.i.d.) samples from a probability distribution. There is the same underlying probability distribution for both the training and test datasets. And each sample is independent of the other samples.

Examples of these assumptions being violated:

• Data is generated by a completely random process such as random walk.
• The training and test dataset come from different probability distributions, either completely different probability distributions or the same probability distribution with different parameters.
• Samples are not independent. An example of dependent samples is dealing cards from a single deck, the probability of later cards is dependent on previously dealt cards.

The model fitting process only has access to the data samples, not the underlying probability distribution. Parametric modeling fitting makes a guess about the functional form of that probability distribution (e.g., Bernoulli or Gaussian) and then, estimates the associated parameters.

Answer 2

The paragraph you mentioned explains a the parametric procedure of creating training data and testing data from a given data set.

Let us take an example let us consider that the distribution of a certain dataset follows Normal distribution (Gaussian)

This means that 68% of the data lies near the mean of the dataset. Also since the dataset has been identified as gaussian we also know the expected probability function (pdf) of the data set assuming we know the mean and variance of the given dataset.

$P(x) = \frac{1}{\sqrt{2 \pi \sigma ^2}} e^{\frac{-(x-\mu)^2}{2 \sigma ^2}}$

Now that we have the formula we can use random variate generation techniques on this formula to create training and test data separately which can be used for the model to learn and test its efficiency.

To learn more about random variate generation I'd direct you to this resource here. It has a great chapter which can help you with understanding the statistical technique behind it.