Prediction in Machine Learning

Question

When we use a regression algorithm in out dataset it's because we assume that there is a relation between our input data and some quantitative value. This is expressed as :

$y = f(x)+\varepsilon $, where $x$ is an input vector and $\varepsilon$ is the random error term.

In the case of a Prediction algorithm, this is the equation :

$\hat{y}=\hat{f}(x)$ where $\hat{f}$ represents our estimate for $f$, and $\hat{y}$ represents the resulting prediction for $y$.

What I don't understand is : why n this setting, $\hat{f}$ is often treated as a black box, in the sense that one is not typically concerned with the exact form of $\hat{f}$, provided that it yields accurate predictions for Y.

Answer 1

Not sure if I can get the direction of your "why", but here's a try:

If you were to use some sort of interpolation, you would have an explicit model of this function in the mathematical sense. This would be a white box, as you actually have the formula/algorithm to do the prediction. If you use a conventional decision tree or linear regression you are still in the same white-box category in the sense that your trained model can be expressed by a manageable set of formulas or rules.

However, in typical machine learning scenarios there are at least two factors that limit the transparency of this box:

1. We care more of the result, and less of the explanation. Thus, we might use different approaches. As long as they fit in the same "interface" as you define it with your function, we don't really care which one is being used. This allows for A/B testing of different approaches and putting in place the one that currently yields best results. After some time, based on new training data or on new advances in machine learning, we can decide to use another algorithm, but the general setup remains the same.
2. Contemporary algorithms are extremely complex, both because the dimensions of the input and output are much higher than our intuition can cope with, and (inherently) because they use many more iterations. Examples of these iterations could be the layers of deep networks, but also the range of algorithms used in ensemble methods like random forests.

Answer 2

A few comments on your questions

What i dont understand is : why n this setting, $\hat{f}$ is often treated as a black box, in the sense that one is not typically concerned with the exact form of $\hat{f}$, provided that it yields accurate predictions for Y .

This is a little inaccurate. The functional form of $f$ (or $\hat{f}$) is of interest, and may algorithms like neural networks do try to approximate $f$ (see Universal approximation theorem). The problem is that many statistical/ML algorithms may not display the approximation in a way that is easily interpretable.

While a neural network may try to approximate the form of $f$, it's architecture is difficult (or nearly impossible) to interpret. Therefore, many people see it as a "black box". Because the shift of interest may be more on prediction, and less on inference of parameter estimates, the fact that a user may have a "black box" is unimportant as long as the main goal of prediction is performed well.

Also, not all statistical/ML algorithms have to be complex like neural networks. You can have simple architectures that work well, and are easily interpretable, like generalized linear models (i.e. linear regression/logistic regression). These algorithms can perform as well as neural networks on certain problems.

So the difference in focus is really just due to the project/research goal, not a difference between statistics and Machine Learning. By the way, there's little difference between the two. The computer science/engineering community has borrowed many ideas from the statistical community long before the term "machine learning" was popular.