I am studying ML and data science stuff from scratch. As a part of the course, I am studying how the models are derived. And for most of them, starting with the simplest - linear regression, we take partial derivatives. I understood its implementation part, however, I am a bit confused about why we need to take partial derivative there.
Is there any specific reason behind it? Can we use any other methodology to compute linear regression loss function?
The underlying idea behind machine learning is to come up with more or less complicated algorithms such that, given a set of input data, one is able to produce some sort of output; this output in turn depends on some parameters (upon which the model is specified). The objective is to choose the aforementioned parameters so that the algorithm output is as close as possible to the actual result. Now let $y_i$ and $f(x_i, \beta)$ be the actual and the predicted value, respectively, (in correspondence of the variable $x_i$): the previous sentence translates into trying to choose the parameters $\beta$ (whatever they mean) such that the error you commit is the smallest, namely $L(y | x, \beta)$ is minimised, where the function $L$ is any way we decide to estimate the error between predictions and actuals. In the literature $L$ is referred to as loss function and for most practical purposes, especially for polynomial models like linear regression, it reduces to the sum of squares $$ L(y|x) = \sum_{i=1}^N (y_i -f(x_i,\beta))^2. $$ For each choice of parameters $\beta$ and function $f(x_i)$ the above takes different values; we are looking for, once we decide to fix the form of $f$, the set of $\beta$ such that the above is the smallest. Assuming the loss function is differentiable, local minima must satisfy the condition such that the collection of partial derivatives with respect to the variable in question (in this case $\beta$) must vanish; as such, at the end of the day one comes down to essentially taking derivatives and equating them to zero.
In case of linear regression one assumes $f(x_i, \beta) = \sum_{j=1}^M x_i^j \beta_j$: plugging this expression into the loss function and taking derivatives gives back the familiar expressions for the coefficients that one learns in school. Likewise for more complicated models: the form of the function $f$ might be more complicated, there might be analytical problems related to the computational minimisation of the loss, there might be a bunch of more parameters connected to each other in a somewhat complex way (for instance for neural networks), nevertheless the underlying argument still holds.
It all comes down on how backward propagation works. Ultimately you need to know how much each part of the equation contributed to the final error and then modify the values of such part of the equation.
In the case of linear regression is a bit more trivial, but when you start stacking one linear regression after other (and that is essentially a neural network) then you need to know how much each coefficient of each layer contributes to the final error. If you were to use a full derivative instead of a partial one, you could not determine the error contribution with such grain-level precision.
I understand this is not a good mathematical explanation, but it should give you an intuition of why you need the partial derivative.
