Probability distribution in input-output pairs

Question

This question might sound silly. But I have been wondering why do we assume that there is a hidden probability distribution between input-output pairs in machine learning setup ?

For example, if we want to learn a function $f: \mathcal{X} \rightarrow \mathcal{Y}$, we generally tend to assume a probability distribution $\rho(x,y)$ on $Z=\mathcal{X} \times \mathcal{Y} $ and try to minimize the error $$ \mathcal{E}(f) = \int (f(x)-y)^2 \ d\rho(x,y) $$

Is the probability distribtution $\rho$ inherent to the very nature of $Z$ or depends on $f$ ?

Can anyone please provide a good intuitive explanation for this ?

Answer 1

Some notes:

1. In the case of supervised learning you assume there is a function $f:\mathcal{X} \rightarrow \mathcal{Y}$, which means there is a connection somehow between inputs and outputs and you want to use it in order to predict. But how this function might look? Without considering the model itself, usually what happens is that there is some noise. And this noise can be in $Y$ and also can be in $X$. This noise gives use the probabilistic setup of the problem, because without it we would have only to solve some equations.

2. Now is important to understand that the noise defines the distribution. So a random variable can be imagined as a function having something fixed and well defined and something not fixed, but taking values according with a distribution. If the variable part does not exist, that you would not have a random variable, right, it would be a simple formula. But its not. Now the $P(X)$ incorporates what happens in $X$ alone, and $P(Y)$ what is in $Y$ alone. When you predict the decision theory says that you are interested in saying which is the most probable value $y_i$ given some input value $x_i$. So you are interested in finding $P(Y|X)$.

3. A joint probability is not always completely described by marginal probabilities. In fact, is completely described only when marginal probabilities are independent. Which means for r.v. $X, Y$ knowing $P(X)$ and $P(Y)$ does not put you in position to know the joint density $P(X, Y)$ (thought for independence you have $P(X,Y)=P(X)P(Y)$).

4. From here you can go directly and try to estimate $P(Y|X)$. In fact if your only interest in only in prediction, this might be a fair bet. A lot of supervised learning algorithms tries to estimate directly this probability. They are called discriminant classifiers. The reason is because they are used to classify something to the class with maximal conditional probability, you discriminate (choose) the maximum value.

5. Now arriving at your question. Notice the obvious $P(X,Y) = P(Y|X)P(X)$ than you see that by trying learning the joint probability, you also learn the conditional (what you need for prediction). This kind of approach is called generative, because knowing the joint probability you can not only predict, but you can generate new data for your problem. More than that, knowing the join probability can give you more insights related with how what are you model works. You can find such kind of information which is not contained only in marginal distributions.

Some final notes:

• Technically you do not minimize the error function, but it's expectation. The error function remains as it is.
• $\mathcal{Z}$ is only a domain it's impossible to describe a probability only by it's domain.

It's late, I hope I was not completely incoherent.

Answer 2

I may still misunderstand what you mean, but the general simple formulation is to minimize sum of loss over all training examples. Converting to your formulation, that 'assumes' the joint distribution of the input and output is just the empirical distribution found in the data. That's the best assumption you can make without additional information. If you had reason to assume something else, you would.