1
$\begingroup$

When we talk about 'Minimizing Loss', we often talk about loss functions such as Mean Squared Error (MSE); the term 'Empirical Risk Minimization' is often used interchangeably. So what's the difference between Error, Loss and Risk?

$\endgroup$
5
$\begingroup$

Error

In this context, error is the difference between the actual / true value ($\theta$) and the predicted / estimated value ($\hat\theta$)

$$Error = \theta - \hat\theta$$

Loss

Loss and Risk are both measurements of how well a model fits the 'data'. The difference is what 'data' means.

Loss ($L$) is a measurement of how well our model perform against your training data. You calculate loss using a loss function, many of which uses error in its equation; for example, simple loss functions may include:

  • Mean Absolute Error (MAE) - average of the errors

    $$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|$$

  • Mean Squared Error (MSE) - average of the squares of the errors

    $$L = \frac{1}{n}\sum_{i=0}^n(\theta - \hat\theta)^2$$

  • Lp - MAE and MSE apply a power of 1 and 2 to the errors, respectively, and averages them. Because of this, they are also called L1 and L2 loss functions. However, you can apply higher (or negative) powers, or $p$, to the errors.

    $$L = \frac{1}{n}\sum_{i=0}^n|\theta - \hat\theta|^p$$

Whilst error is simply difference between $\theta$ and $\hat\theta$, loss can differ depending on which loss function you pick. The loss function you pick depends on your data and your objectives. For instance, MAE, or L1, are sensitive to outliers; so if you want your model to also be sensitive to outliers, then you can use MAE.

Your loss depends on the loss function used, which depends on your use case; on the other hand, the error is always the same.

Risk

To re-iterate, loss measures how well your model fits against your training data. However, our end goal is not to fit our model to our training data, which can lead to overfitting. Instead, it is to fit against our validation and test data, or simply any new unseen data.

This is where (true) risk comes in. Risk is the average measure of loss, or expected loss, across your whole data distribution.

$$R(\theta,\hat\theta) = \mathbb{E}[L(\theta,\hat\theta)]$$

To illustrate, let's imagine that you have an overfitted model. Both the errors and loss of your model would be very low (because they are measured against your training data). But because it is overfitted, you'd expect the loss for any new data to be high. Thus, you have a model which has low error and loss, but very high risk.

Empirical Risk

To bring this back full circle - when we train our model, we do not have the full distribution of the data. This may be because some of our data is used for validation and testing, or that new data points are produced in real-time. The best we can do is to pick our training data in a random way and assume that our training data is representative of the real data.

Therefore, because we don't have all the data, the best we can do is to minimize the empirical risk, from data that we do have (our training data), and use regularization techniques to generalize (i.e. avoid overfitting). This is why minimizing loss and minimizing empirical risk are roughly the same thing.

Summary

When we are training our model, our focus should not be on minimizing errors or loss, but to minimize true risk. But most of the time, we can't, so we minimize the empirical risk and regularize.

$\endgroup$
  • $\begingroup$ What do you mean by "In this context, error is [...]"? What is this context? Machine learning in general? What is a context in which the error is not just the difference between true values $\theta$ and the predicted values $\hat{\theta}$? I am asking because I want to know if in ML there are also other possible definitions of error, that are not just the difference $\theta - \hat{\theta}$. $\endgroup$ – lo tolmencre Jul 28 at 10:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.