6
$\begingroup$

I have a question about regression loss functions like Mean Absolute Error (MAE) and Mean Squared Error (MSE) used in deep learning.

When we calculate these losses, we remove the plus/minus sign from the error (predicted value - true value). We just take the absolute value or square of the raw number.

For example, let's say the true value is 3. If we predict 2, the error is -1. If we predict 4, the error is +1. But when we compute MAE or MSE, we get 1 for both cases because the plus/minus sign is removed.

My question is, how can the model know whether to increase or decrease its weights and biases? For a -1 error, it should increase weights/biases. For a +1 error, it should decrease them. But the loss itself has removed this plus/minus information.

I understand backpropagation calculates gradients to update weights. But I'm confused how the model "remembers" the plus/minus sign during this process, since the loss function itself has removed it.

Can you explain in simple terms how deep learning models can learn correctly using MAE or MSE, even though the plus/minus sign is removed when computing the loss? Any insights on how this sign information is kept would be really helpful. Thanks!

$\endgroup$

2 Answers 2

7
$\begingroup$

You're absolutely right that these loss functions seemingly discard the sign information by taking the absolute value. The key insight, however, is that while the loss function itself discards the sign, the gradients calculated during backpropagation still contain the sign information from the original errors. Here's my version of a step-by-step breakdown of how this works:

  1. During the forward pass, we make predictions and calculate the raw errors (predicted - true) for each example. These raw errors still have their original +/- signs.
  2. We compute the loss (MAE or MSE) by squaring these raw errors. This is where the sign info seems to be lost.
  3. During backpropagation, we calculate the gradients of the loss with respect to the weights using the chain rule from calculus.
  4. Crucially, these gradients depend on the derivative of the loss function with respect to the model's predictions. And this derivative encodes the original sign information.
  5. For example, the derivative of the squared error $(y_{pred} - y_{true})^{2}$ with respect to y_pred is $2*(y_{pred} - y_{true})$. This derivative is positive if $y_{pred} > y_{true}$ (telling us to decrease weights) and negative if $y_{pred} < y_{true}$ (telling us to increase weights).
  6. So while the loss itself has discarded the signs, the gradients used to actually update the weights still "remember" whether the error was positive or negative for each example.
  7. During gradient descent, we update the weights using these gradients, which contain the crucial sign information allowing the model to learn correctly.
$\endgroup$
3
$\begingroup$

But the loss itself has removed this plus/minus information.

From the loss value, you can't tell which direction to go in. However, we also have the loss gradient, which does include directional information.

The MSE loss is:

$\mathscr{L}_{MSE} = \frac{1}{n}\sum\limits_{i=0}^{n-1}error_i^{~2} =\frac{1}{n}\sum(y_i-\hat{y_i})^2$

This yields a number that doesn't take signs into account. Suppose the model is $\hat{y}=\theta_1x+\theta_0$.

The slope of the loss with respect to the model's parameters is:

$\frac{\partial MSE}{\partial\theta_1}=-\frac{2}{n}\sum (y_i-\hat{y}_i)x_i=-\frac{2}{n}\sum error_i \times x_i$

$\frac{\partial MSE}{\partial\theta_0}=-\frac{2}{n}\sum (y_i-\hat{y}_i)~~~~=-\frac{2}{n}\sum error_i$

By defining the loss as the MSE, the gradient terms don't lose the sign information as they use $error_i$ directly. The update is done using the gradients:

$\theta:=\theta - \eta\frac{\partial MSE}{\partial \theta}$

Thus, the update takes the sign of the error into account, as well as its size.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.