3
$\begingroup$

I am reading the article Stochastic Gradient Descent Tricks by Léon Bottou (avaible here) and on the very first page they introduce empirical risk

$E_n(f) = \frac{1}{n} \sum_{i=1}^{n} l(f(x_i),y_i),$

where $l(f(x),y)$ is a loss function, that measures the cost of predicting $f(x)$ when the actual answer is $y$.

Then, there is written:

We seek the function $f \in \mathcal{F}$ that minimizes the loss $Q(z,w) = l(f_w(x),y)$ averaged on the examples.

I fail to see the difference between empirical loss and the $Q$ loss function, can anybody provide explanation? Also I am not native speaker so there might be misunderstanding in "averaged on examples".

$\endgroup$
1
$\begingroup$

The loss function is the function used to measure the quality of the approximation $f$. On the other hand, the empirical risk is a function that results from averaging the loss function over your data.

More formally, consider that your data is drawn from a set $\Omega$ and let $\mathcal{D}$ be the set of all the possible functions $f$ that you can choose. Then the loss function is a function $L\colon\Omega\times\mathcal{D}\to\mathbb{R}_{+}$. If $\{\omega_i\}_{i\in I}\subseteq\Omega$ is a finite family and $L$ is a loss function, then the empirical risk associated with each element $f\in\mathcal{D}$ is calculated as $$\rho(f)=\frac{1}{\vert I \vert}\sum_{i\in I}L(\omega_i,f).$$

Note that you got confused with the domains while writing the question. Let me clarify that for you:

In the situation you described, considering that each of your 'data points' $(x,y)$ belongs to a set $X\times Y$, we obtain $\Omega=X\times Y$, and then you have that $Q\colon X\times Y\times \mathcal D\to\mathbb{R}_{+}$ is a function given by $$Q(x,y,f)=l(f(x),y).$$

Also, the empirical risk becomes $$\frac{1}{\vert I\vert}\sum_{i\in I}l(f(x_i),y_i)=\frac{1}{\vert I\vert}\sum_{i\in I}Q(x_i,y_i,f).$$

Moreover, I should warn you that dividing the sum by the size of your data set does not change the optimal function.

Hope this helps!

$\endgroup$
  • $\begingroup$ The answer by Ariel Serranoni basically addresses your question. I only like to add that basically: the loss function is the function you get by measuring the difference on all data you have, while the empirical loss function is what you get when you measure the difference on only a small subset of the data (for example: training set in DNN) which you use to train your predictor. For a comprehensive reference, you can consult: S. Shalev-Shwartz and S. Ben-David, Understanding machine learning: from theory to algorithms, Cambridge University Press, 2014. $\endgroup$ – Tuyen May 19 '19 at 4:37
1
$\begingroup$

I found the accepted answer pretty hard to understand. Here is a simplified version that cleared it up for me:

Loss Function: a loss or risk function, $\mathcal{L}(\hat{y}^i, y^i)$, quantifies how well (more accurately, how badly) $\hat{y}$ approximates y. smaller values of $\mathcal{L}(\hat{y}^i, y^i)$ indicate that $\hat{y}$ is a good approximation of $y$

Empirical Risk: the empirical risk is the average loss over the data points. $$\mathcal{L} = \frac{1}{n}\sum_{i=1}^n \mathcal{L}(\hat{y}^i, y^i)$$

See these Stanford Lecture Slides for more details.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.