I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk

$E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$

where $l(f(x),y)$ is a loss function, that measures the cost of predicting $f(x)$ when the actual answer is $y$.

Then, there is written:

We seek the function $f \in \mathcal{F}$ that minimizes the loss $Q(z,w) = l(f_w(x),y)$ averaged on the examples.

I fail to see the difference between empirical loss and the $Q$ loss function, can anybody provide explanation? Also I am not native speaker so there might be misunderstanding in "averaged on examples".


2 Answers 2


I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:

Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how well (more accurately, how badly) $\hat{y}$ approximates y. smaller values of $\mathcal{L}(\hat{y}^i, y^i)$ indicate that $\hat{y}$ is a good approximation of $y$

Empirical Risk: the empirical risk is the average loss over the data points. $$\mathcal{L} = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(\hat{y}^i, y^i)$$

See these Stanford Lecture Slides for more details.

  • $\begingroup$ What does the index i indicate? Why are there several loss functions? $\endgroup$
    – Ben
    Commented Jun 10, 2021 at 12:34
  • 1
    $\begingroup$ @Ben the index i is used to iterate over each of the training examples. There is a single loss function but you calculate the loss separately for each training example and then average them all to find the empirical risk. $\endgroup$ Commented Jun 10, 2021 at 16:05
  • 1
    $\begingroup$ Thank you! So, basically, the empirical risk is the average loss (as visible above). $\endgroup$
    – Ben
    Commented Jun 14, 2021 at 10:57
  • $\begingroup$ @Ben yep exactly $\endgroup$ Commented Jun 14, 2021 at 18:04
  • $\begingroup$ a term like "average loss" would have been much easier, like in "root mean square" or so :) But thanks a lot for clarifying! $\endgroup$
    – Ben
    Commented Jun 15, 2021 at 6:44

The loss function is the function used to measure the quality of the approximation $f$. On the other hand, the empirical risk is a function that results from averaging the loss function over your data.

More formally, consider that your data is drawn from a set $\Omega$ and let $\mathcal{D}$ be the set of all the possible functions $f$ that you can choose. Then the loss function is a function $L\colon\Omega\times\mathcal{D}\to\mathbb{R}_{+}$. If $\{\omega_i\}_{i\in I}\subseteq\Omega$ is a finite family and $L$ is a loss function, then the empirical risk associated with each element $f\in\mathcal{D}$ is calculated as $$\rho(f)=\frac{1}{\vert I \vert}\sum_{i\in I}L(\omega_i,f).$$

Note that you got confused with the domains while writing the question. Let me clarify that for you:

In the situation you described, considering that each of your 'data points' $(x,y)$ belongs to a set $X\times Y$, we obtain $\Omega=X\times Y$, and then you have that $Q\colon X\times Y\times \mathcal D\to\mathbb{R}_{+}$ is a function given by $$Q(x,y,f)=l(f(x),y).$$

Also, the empirical risk becomes $$\frac{1}{\vert I\vert}\sum_{i\in I}l(f(x_i),y_i)=\frac{1}{\vert I\vert}\sum_{i\in I}Q(x_i,y_i,f).$$

Moreover, I should warn you that dividing the sum by the size of your data set does not change the optimal function.

Hope this helps!

  • $\begingroup$ The answer by Ariel Serranoni basically addresses your question. I only like to add that basically: the loss function is the function you get by measuring the difference on all data you have, while the empirical loss function is what you get when you measure the difference on only a small subset of the data (for example: training set in DNN) which you use to train your predictor. For a comprehensive reference, you can consult: S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: from theory to algorithms, Cambridge University Press, 2014. $\endgroup$
    – Tuyen
    Commented May 19, 2019 at 4:37

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.