I am currently wondering if the following graphic of deep residual networks is wrong:

enter image description here

I would say the graphic describes

$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$

The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?

In the text, they define

$$\mathcal{F}(x) := \mathcal{H}(x) - x$$

where $\mathcal{H}(x)$ is "the desired underlying mapping" (whatever that exactly means).

Also, equation (1) seems strange to me: $$y = \mathcal{F}(x, \{W_i\}) + x$$ In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?

My thoughts

I think the authors could mean $$\mathcal{F}_i = \varphi(W_i x)$$ In that case, in the image where $\mathcal{F}(x)$ is it should be $$\mathcal{F}_1(x) = \varphi(W_1 x)$$ and where $\mathcal{F}(x) + x$ is should be $$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$


(Per the diagram), $F(x)$ here is simply the entire two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it!

The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps!

PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an entire deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation.


(edited) answer of /u/mostly_reasonable on reddit

The thing to note here is that $F(x)$ can refer to the functioning of more than one layer. The paper's authors use '$H(x)$' to mean something like 'the function we want to learn in some (possibly more than one) consecutive layers of a neural network', see their statement

[...] hoping each few stacked layers directly [...]

Then '$F(x)$' is then that same possibly multi layer function, minus the residuals. The author of course hypothesizes that $F(x)$ is easier to learn than $H(x)$. So I think that in the figure the $F(x)$ is supposed to refer to everything in the Figure besides the ('$+ x$') part. Note how the F(x) symbol is centered with respect the the network, not attached to either layer. Then $F(x) + x$ references the entire $F(x)$ two layer network above combined with the skip connections.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.