8
$\begingroup$

A recent paper by He et al. (Deep Residual Learning for Image Recognition, Microsoft Research, 2015) claims that they use up to 4096 layers (not neurons!).

I am trying to understand the paper, but I stumble about the word "residual".

Could somebody please give me an explanation / definition what residual means in this case?

Examples

We explicitly reformulate the layers as learning residual functions with reference to the layer inputs, instead of learning unreferenced functions.

[...]

Instead of hoping each few stacked layers directly fit a desired underlying mapping, we explicitly let these layers fit a residual mapping. Formally, denoting the desired underlying mapping as $\mathcal{H}(x)$, we let the stacked nonlinear layers fit another mapping of $\mathcal{F}(x) := \mathcal{H}(x)−x$. The original mapping is recast into $\mathcal{F}(x)+x$. We hypothesize that it is easier to optimize the residual mapping than to optimize the original, unreferenced mapping

Improve this question
$\endgroup$
1
  • $\begingroup$ This might be a language problem. If you know the German translation of "residual" in this context, I would be happy about it, too. $\endgroup$
    – Martin Thoma
    Jan 24, 2016 at 16:49

1 Answer 1

Reset to default
3
$\begingroup$

It's $F(x)$; the difference between the mapping $H(x)$ and its input $x$. It's a common term in mathematics (DE).

Improve this answer
$\endgroup$
2
  • 8
    $\begingroup$ This is not correct. The term Residual, as is found in mathematics, is not the same as the residual mapping the paper talks about. Per the link you've listed, we see that for f(x)=b, the residual is the difference b-f(x). The residual mapping is per their definition the difference between the input x and the output of the function H(x). $\endgroup$
    – spurra
    Jan 23, 2017 at 10:29
  • $\begingroup$ @spurra Is there a reason why the authors named the thing "residual" networks when it isn't strictly the term Residual found in mathematics? Is it because the authors wanted to emphasize that when the desired mapping (H(x)) is an identity mapping (H(x)=x), F(x)=H(x)-x becomes the residual? $\endgroup$
    – Danny Han
    Nov 9, 2021 at 3:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.