# Is the graphic of deep residual networks wrong?

I am currently wondering if the following graphic of deep residual networks is wrong: I would say the graphic describes

$$\varphi \left (W_2 \varphi(W_1 x) + x \right ) \qquad \text{ with } \varphi = ReLU$$

The $\mathcal{F}(x)$ does not make sense to me. Assuming both weight layers are simple MLPs without bias, where the first one has a weight matrix $W_1$ and the second one has a weight matrix $W_2$, what is $\mathcal{F}$?

In the text, they define

$$\mathcal{F}(x) := \mathcal{H}(x) - x$$

where $\mathcal{H}(x)$ is "the desired underlying mapping" (whatever that exactly means).

Also, equation (1) seems strange to me: $$y = \mathcal{F}(x, \{W_i\}) + x$$ In figure 5 they have two weight layers and call this a building block. Why is there only one weight matrix in this equation?

## My thoughts

I think the authors could mean $$\mathcal{F}_i = \varphi(W_i x)$$ In that case, in the image where $\mathcal{F}(x)$ is it should be $$\mathcal{F}_1(x) = \varphi(W_1 x)$$ and where $\mathcal{F}(x) + x$ is should be $$\mathcal{F}_2(\mathcal{F}_1(x)) + x = \varphi \left (W_2 \varphi(W_1 x) + x \right )$$

## 2 Answers

(Per the diagram), $F(x)$ here is simply the entire two-layer non-linear chain that is operating on the input $x$. Then, the final output is simply $F(x) + x = H(x)$. That's it!

The thing that may be confusing you is that $F(.)$. In this case, they do not mean for $F$ to simply encompass one operation. Instead, it encompasses any set of operations processing $x$, up until you add $x$ back. Hope that helps!

PS: It is also common to see this type of nomenclature in a lot of DNN literature, whereby one refers to an entire deep non-linear chain as $D(x)$. For example in Generative Adversarial Networks, (GAN)s, $D(x)$ refers to the entire deep net devoted to the discrimination process, while $G(x)$ refers to the entire net devoted to the noise shaping. In both cases, they are composed of entire functions/nets, and do not signify simply one operation.

(edited) answer of /u/mostly_reasonable on reddit

The thing to note here is that $F(x)$ can refer to the functioning of more than one layer. The paper's authors use '$H(x)$' to mean something like 'the function we want to learn in some (possibly more than one) consecutive layers of a neural network', see their statement

[...] hoping each few stacked layers directly [...]

Then '$F(x)$' is then that same possibly multi layer function, minus the residuals. The author of course hypothesizes that $F(x)$ is easier to learn than $H(x)$. So I think that in the figure the $F(x)$ is supposed to refer to everything in the Figure besides the ('$+ x$') part. Note how the F(x) symbol is centered with respect the the network, not attached to either layer. Then $F(x) + x$ references the entire $F(x)$ two layer network above combined with the skip connections.