What does it mean for the training data to be generated by a probability distribution over datasets

Question

I was reading the Deep Learning book and came across the following para (page 109, second para):

The training and test data are generated by a probability distribution over datasets called the data-generating process. We typically make a set of assumptions known collectively as the i.i.d. assumptions. These assumptions are that the examples in each dataset are independent from each other and that the training set and test set are identically distributed, drawn from the same probability distribution as each other. This assumption enables us to describe the data-generating process with a probability distribution over a single example.The same distribution is then used to generate every train example and every test example. We call that shared underlying distribution the data-generating distribution, denoted $p_{\text{data}}$. This probabilistic framework and the i.i.d. assumptions enables us to mathematically study the relationship between training error and test error.

Can somebody please explain to me the meaning of this paragraph?

On page 122 the last paragraph, it also gives an example

a set of samples $\{x(1), \dots, x(m) \}$ that are independently and identically distributed according to a Bernoulli distribution with mean $\theta$.

What does this mean?

Answer 1

There is a common assumption that data that is being modeled is independent and identically distributed (i.i.d.) samples from a probability distribution. There is the same underlying probability distribution for both the training and test datasets. And each sample is independent of the other samples.

Examples of these assumptions being violated:

• Data is generated by a completely random process such as random walk.
• The training and test dataset come from different probability distributions, either completely different probability distributions or the same probability distribution with different parameters.
• Samples are not independent. An example of dependent samples is dealing cards from a single deck, the probability of later cards is dependent on previously dealt cards.

The model fitting process only has access to the data samples, not the underlying probability distribution. Parametric modeling fitting makes a guess about the functional form of that probability distribution (e.g., Bernoulli or Gaussian) and then, estimates the associated parameters.

Answer 2

The paragraph you mentioned explains a the parametric procedure of creating training data and testing data from a given data set.

Let us take an example let us consider that the distribution of a certain dataset follows Normal distribution (Gaussian)

This means that 68% of the data lies near the mean of the dataset. Also since the dataset has been identified as gaussian we also know the expected probability function (pdf) of the data set assuming we know the mean and variance of the given dataset.

$P(x) = \frac{1}{\sqrt{2 \pi \sigma ^2}} e^{\frac{-(x-\mu)^2}{2 \sigma ^2}}$

Now that we have the formula we can use random variate generation techniques on this formula to create training and test data separately which can be used for the model to learn and test its efficiency.

To learn more about random variate generation I'd direct you to this resource here. It has a great chapter which can help you with understanding the statistical technique behind it.

Answer 3

When you first encounter machine learning methods, or statistics in general, you are often directly presented with a sample from a data set $D = \{ (x_1, y_1), (x_2,y_2), \cdots, (x_n,y_n)\}$.

The general paradigm which ignores theory is that you then split into training/testing data, train your model, then evaluate on testing data. What many practitioners do not think about, but that is important for understanding theory, but even more importantly for understanding your data, is what is the data generating process.

The data $D$ is generally assumed to be $n$ samples from some joint distribution $P(X,Y)$ satisfying certain assumptions (below). As a concrete example, we could imagine your data is generated as follows: $$\begin{align} X &\sim \textrm{Unif}[0,1]\\ Y &= X + \epsilon \\ \epsilon &\sim N(0,1)\end{align}$$

In this case, points of $X$ are sampled from the standard uniform distribution, $Y$ is generated by applying the linear function + noise. We of course do not know the data generating process ahead of time but we make certain assumptions about it that Brian's answer above already describes: eg. data is I.I.D and training and testing data come from the same distribution. If these assumptions were false (eg. train and test data are different distributions) then our model trained on training data would have no predictive value on testing data. Our goal in modeling the data is to try to find the relationship between $Y$ and $X$ when the equations above are not privy to us (and probably don't even exist in some analytical form).

Assuming our data comes from a fixed reference distribution $P(X,Y)$ is actually a strong assumption, and it can be confusing for people new to the field to think through. It is reasonable to ask - why should my data be generated by some fixed reference distribution at all? How can I verify this assumption? I won't go into more details here, but I bring this up just to point out that the assumptions here aren't "obvious" and requires very careful consideration by the model builder. Let's just say that both the theory and application of ML in practice would fail if this wasn't true.

Concrete Practical Example: A common issue that practitioners in industry must deal with is Covariate Shift where variables shift over time.

• For example, a model trained for facial recognition may be trained on predominantly male faces, and then fail to generalize to females when applied to the general population. In this case $P_{\textrm{train}}(X,Y) \neq P_{\textrm{test}}(X,Y)$. This is one of the reasons models need to be trained on samples representative of the population in which it will be used.
• Another example is considering a model that is used to predict if someone will click on an ad. If this model is then used to generate new ads that are shown to users, it is actually changing the underlying distribution of the data generating process itself. This is a common challenge faced in deploying ML models in production and there are many resources available for you to read about it.

This answer is far from complete but hopefully helps point you in the right direction for thinking through these questions.