Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.
Sign up to join this community
What is the purpose of computing the partial derivative of the loss function in order to find the best parameters that minimize the error?
Considering the loss function of a linear model, we want to find the best parameters that minimize the error. I do not understand why this result can be achieve considering the parameters with respect to which the partial derivative (with respect to each parameter) of the loss function is equal to 0.
It’s a minimization problem. The typical calculus approach is to find where the derivative is zero and then argue for that to be a global minimum rather than a maximum, saddle point, or local minimum.
In a nice situation like linear regression with square loss (like ordinary least squares), the loss, as a function of the estimated parameters, is quadratic and up-opening. Thus, when we find a point with a derivative of zero, it is assured to be a global minimum.
Therefore, start taking the partial derivatives and finding where they equal zero.
EXAMPLE (SIMPLE LINEAR REGRESSION)
$$ \hat y=\hat\beta_0+\hat\beta_1x\\ L(y,\hat\beta_0,\hat\beta_1)=\sum_{i=1}^N\bigg( y_i - \hat\beta_0-\hat\beta_1x_i \bigg)^2 $$
Now solve as system of equations for the optimal $\hat\beta_0$ and $\hat\beta_1$.
$$ \dfrac{\partial L}{\partial \hat\beta_0}=0\\ \dfrac{\partial L}{\partial \hat\beta_1}=0 $$
There is a geometric argument for why the solution is a global minimum, but it might be worth doing once the entire second-derivative test from multivariable calculus, just to see how it all works.
First thing to do is make a clear distinction between loss and error. The loss function is the function an algorithm minimizes to find an optimal set of parameters during training. The error function is used to assess the performance this model after it has been trained. We always minimize loss when training a model, but this won't neccessarily result in a lower error on the train or test set. In fact, you could argue that we are deliberatly increasing error on the train set when we are trying to prevent overfitting. Obviously we would ideally like to minimize error on the test set, but this isn't the same as minimizing loss. Minimizing the error involves selecting good features, the appropriate model, fine tuning hyperparameters, etc. whereas loss is minimized during training, regardless of what features or model you use.
Now I'll try to address you're question on partial derivatives in 2 parts. One part telling what are the criteria to find a minimum and another part to explain why these are the criteria to find a minimum
What are the criteria to find the minimum of a loss function
Suppose you have a loss function $L(\omega)$ which is a function of the parameters
$$\omega= \begin{pmatrix} \omega_{1}\\ \omega_{2}\\ \vdots \\ \omega_{N} \end{pmatrix} $$
The local minima of such a function are found by finding values of $\omega_j$ which satify the following
\begin{equation} \nabla L = 0 \end{equation}
and \begin{equation} det(H_L) \ge 0 \end{equation} where $\nabla L$ denotes the gradient of $L$ and $det(H_L)$ denotes the determinant of the Hessian matrix of second partial derivatives of $L$ This wiki page has further details https://en.wikipedia.org/wiki/Hessian_matrix#Second-derivative_test
However, it's generally assumed that the function $L(\omega)$ is a convex function(most loss functions you'll see are in fact convex). One of the properties of convex functions is that $det(H_L)\ge0$. So for convex functions we only need to solve for $\nabla L = 0$
Wiki page on convex funtions: https://en.wikipedia.org/wiki/Convex_function#Functions_of_several_variables
Why are these the criteria to find the minimum of a loss function?
But another question you might have from reading this is:
"You've told me how to find a minimum, but you haven't told me why these criteria find a minimum"
To think about this remember what $\nabla L$ tell us about the function $L$. $\nabla L$ tells us about the "steepness" of $L$, i.e. the vector $\nabla L$ points in the direction where $L$ is steepest and the norm of $\nabla L$ tells you how steep $L$ is. So when $\nabla L=0$ for some value of $\omega$ it means, $L$ is "flat" in every direction for that value of $\omega$.
As for $det(H_L)$, this value (kind of) tells you how $\nabla L$ itself changes. So when $det(H_L)\ge0$ it mean $\nabla L$ will increase in every direction. But based on what we said previously, if $\nabla L$ is increasing in every direction, then the "steepness" of $L$ will increase in every direction.
So let's put these 2 observations together. If $\nabla L = 0$ and $det(H_L)\ge0$ for some value of $\omega$ it means for that particular value of $\omega$, $L$ is flat but the steepness of $L$ will go up if you move in any direction. So $L$ must form a sort of bowl shape where $\omega$ identifies the bottom of the bowl. But the bottom of the bowl is the minimum value of $L$. Therfore $\omega$ is where $L$ is minimized.
