Is empirical risk the same thing as loss function?

Question

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk

$E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$

where $l(f(x),y)$ is a loss function, that measures the cost of predicting $f(x)$ when the actual answer is $y$.

Then, there is written:

We seek the function $f \in \mathcal{F}$ that minimizes the loss $Q(z,w) = l(f_w(x),y)$ averaged on the examples.

I fail to see the difference between empirical loss and the $Q$ loss function, can anybody provide explanation? Also I am not native speaker so there might be misunderstanding in "averaged on examples".

Answer 1

I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:

Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how well (more accurately, how badly) $\hat{y}$ approximates y. smaller values of $\mathcal{L}(\hat{y}^i, y^i)$ indicate that $\hat{y}$ is a good approximation of $y$

Empirical Risk: the empirical risk is the average loss over the data points. $$\mathcal{L} = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(\hat{y}^i, y^i)$$

See these Stanford Lecture Slides for more details.

Answer 2

The loss function is the function used to measure the quality of the approximation $f$. On the other hand, the empirical risk is a function that results from averaging the loss function over your data.

More formally, consider that your data is drawn from a set $\Omega$ and let $\mathcal{D}$ be the set of all the possible functions $f$ that you can choose. Then the loss function is a function $L\colon\Omega\times\mathcal{D}\to\mathbb{R}_{+}$. If $\{\omega_i\}_{i\in I}\subseteq\Omega$ is a finite family and $L$ is a loss function, then the empirical risk associated with each element $f\in\mathcal{D}$ is calculated as $$\rho(f)=\frac{1}{\vert I \vert}\sum_{i\in I}L(\omega_i,f).$$

Note that you got confused with the domains while writing the question. Let me clarify that for you:

In the situation you described, considering that each of your 'data points' $(x,y)$ belongs to a set $X\times Y$, we obtain $\Omega=X\times Y$, and then you have that $Q\colon X\times Y\times \mathcal D\to\mathbb{R}_{+}$ is a function given by $$Q(x,y,f)=l(f(x),y).$$

Also, the empirical risk becomes $$\frac{1}{\vert I\vert}\sum_{i\in I}l(f(x_i),y_i)=\frac{1}{\vert I\vert}\sum_{i\in I}Q(x_i,y_i,f).$$

Moreover, I should warn you that dividing the sum by the size of your data set does not change the optimal function.

Hope this helps!