Huber loss formula is

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ \delta (|a| - \frac{1}{2} \delta) && |a| > \delta\end{cases}$ where $a = y - f(x)$

As I read on Wikipedia, the motivation of Huber loss is to reduce the effects of outliers by exploiting the median-unbiased property of absolute loss function $L(a) = |a|$ while keeping the mean-unbiased property of squared loss function $L(a) = a^2$ within the typical region (i.e. region without outliers)

I don't know why don't Huber loss has the form:

$\hspace{3.0cm} L_\delta(a) = \begin{cases} \frac{1}{2} a^2 && |a| \leq \delta \\ c |a| && |a| > \delta\end{cases}$ where $c$ is some constant which controls the error magnitude of outliers

or similar forms ?


Let's check for continuity for the new proposed function. For convenience, let me call it $L_{\delta, c}$.

$$\lim_{a \to \delta^+}L_{\delta, c}(a)=c\delta$$ and $$\lim_{a \to \delta^-}L_{\delta, c}(a)=\frac{\delta^2}2$$

If we want the function to be continuous, we would want $c$ to take value $\frac{\delta}2$.

Now, let's check for diffentiability of $L_{\delta, \frac{\delta}2}$.

$$\lim_{h \to 0^+} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^+} \frac{\frac{\delta(\delta+h)}2-\frac{\delta^2}2}{h}=\frac{\delta}2$$

$$\lim_{h \to 0^-} \frac{L_{\delta, \frac{\delta}2}(\delta+h)-L_{\delta, \frac{\delta}2}(\delta)}{h}=\lim_{h \to 0^-} \frac{\frac{(\delta+h)^2}2-\frac{\delta^2}2}{h}=\delta$$

which shows that we can't have $L_{\delta,c}$ that is differentiable.

Now, let's examine the differentiable of the original huber loss at $\delta$.

$$\lim_{a \to \delta^+}L_\delta (a)=\delta(\delta-\frac{\delta}2)=\lim_{a \to \delta^-}L_{\delta}(a)=L_{\delta}(\delta)$$


$$\lim_{h \to 0^+} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}=\lim_{h \to 0^+} \frac{\delta(\frac\delta2+h)-\frac{\delta^2}2}{h}=\delta=\lim_{h \to 0^-} \frac{L_{\delta}(\delta+h)-L_{\delta}(\delta)}{h}.$$

The huber loss function is constructed such that it is differentiable.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.