Is any feed-forward neural network a composable function?

Question

I am wondering if each feed-forward neural network can be written as a composition of the layer functions.

So if $F(x)$ is the output of a neural network F, do always exists layer functions $f_1$,...,$f_n$ with $F(x)$ = ($f_1$ o .. o $f_n$)$(x)$? Here "o" denotes the composition symbol.

If the network consists only of layers where each layer is connected to exactly one previous layer, this is obvious. However, this is not always the case, e.g. if you have skip-connections.

So what about the "concat" layer and the "sum" layer?

So, what if $F(x)$ = $G_1(x) + G_2(x)$, or $F(x) = [H_1(x), H_2(x)]$, for some layer functions $G_1$,$G_2$,$H_1$,$H_2$? Can $F$ still be written as a composition?

Answer 1

I suppose you could write $ h\circ [g\circ f\ |\ f]$ if you argue that matrix augmentation is itself a multivariate function.

But to avoid pedantic arguments, you could use multivariate function composition (although it is uglier).

Define a multivariate concatenate function for column vectors: $ c(\vec{x_1},\vec{x_2},...,\vec{x_n})=\left[ \begin{matrix} \vec{x_1} \\ \vec{x_2} \\ \vdots \\ \vec{x_n} \end{matrix} \right]$

In a network $f\rightarrow g\rightarrow h$ having skip connection $f\rightarrow h$, you could then write: $ h\circ c|_{g\circ f,f}$.