How do I determine if a neural network can be computed by a network that contains only an input layer and an output layer?

Question

I have a homework problem where the neural network below is given with its description. We have been asked "Can the functions that the above network computes also be computed by a network that contains only an input layer and an output layer that has a single node?" If we can, we're to provide the network, weights, and activation function. And if not, we have to explain why.

I am very unsure how to approach this problem as I am not that strong with neural networks. Any starting point would be very helpful.

Thanks in advance.

Answer 1

You are asked whether we can mimic the given neural network with a new network that contains only a input layer and an output layer, which in turn means whether $a_5$ can be represented as a linear combination of $x_1, x_2, x_3, x_4$. Since every action the neural net does is linear it can be represented as a smaller one without a hidden layer.

Let's calculate the weights

$$a_5 = (a_1*\theta_9 + a_2*\theta_10 + a_3*\theta_11 + a_4*\theta_12)*C $$ $$a_5 = ((x_1*\theta_1 + x_2*\theta_2)*\theta_9*C + (x_1*\theta_3 + x_2*\theta_4)*\theta_{10}*C + (x_3*\theta_5 + x_4*\theta_6)*\theta_{11}*C+ (x_3*\theta_7 + x_4*\theta_8)*C)*\theta_{12}*C $$ $$a_5 = C^2 * (x_1*(\theta_1*\theta_9+\theta_3*\theta_{10}) + x_2(\theta_2*\theta_{9}+\theta_4*\theta_{10}) + x_3(\theta_5*\theta_{11}+ \theta_7*\theta_{12}) + x_4(\theta_6*\theta_{11}+\theta_8*\theta_{12})) $$

So our weights are: $$<\theta_1*\theta_9+\theta_3*\theta_{10},\theta_5*\theta_{11}+ \theta_7*\theta_{12} , \theta_2*\theta_{9}+\theta_4*\theta_{10}, \theta_6*\theta_{11}+\theta_8*\theta_{12}> $$ and our activation function is multiplication by $C^2$

Answer 2

Yes, it is possible to represent the neural network with a single hidden layer. Computing the final result of $a_5$:

$a_5 = (\theta_9a_1+\theta_{10}a_2+\theta_{11}a_3+\theta_{12}a_4)*C$ (Equation 1)

$a_1 = (\theta_1x_1+\theta_2x_2)*C$ (Equation 2)

$a_2 = (\theta_3x_1+\theta_4x_2)*C$ (Equation 3)

$a_3 = (\theta_5x_3+\theta_6x_4)*C$ (Equation 4)

$a_4 = (\theta_7x_3+\theta_8x_4)*C$ (Equation 5)

Replacing Equations 2,3,4,5 into Equation 1:

$$a_5=(\theta_9*(\theta_1x_1+\theta_2x_2)*C+\theta_{10}*(\theta_3x_1+\theta_4x_2)*C+\theta_{11}*(\theta_5x_3+\theta_6x_4)*C+\theta_{12}*(\theta_7x_3+\theta_8x_4)*C)*C$$

$$a_5=(\theta_9\theta_1x_1+\theta_9\theta_2x_2+\theta_{10}\theta_3x_1+\theta_{10}\theta_4x_2+\theta_{11}\theta_5x_3+\theta_{11}\theta_6x_4+\theta_{12}\theta_7x_3+\theta_{12}\theta_8x_4)*C^2$$9 $$a_5 = (\theta_9\theta_1+\theta_{10}\theta_3)C^2x_1+(\theta_9\theta_2+\theta_{10}\theta_4)C^2x_2+(\theta_{11}\theta_5+\theta_{12}\theta_7)C^2x_3+(\theta_{11}\theta_6+\theta_{12}\theta_8)C^2x_4$$

Calling $F(\theta,x) = C*\sum\theta x$ the activation function (according to the definition), we reexpress the above equation with this definition:

$$a_5 = F([\theta_9\theta_1+\theta_{10}\theta_3,\theta_9\theta_2+\theta_{10}\theta_4,\theta_{11}\theta_5+\theta_{12}\theta_7,\theta_{11}\theta_6+\theta_{12}\theta_8],[x_1,x_2,x_3,x_4])*C$$

Which means we can define the result of the network with only one equation which depends on the activation function $F$.

The network is (I can't draw here, sorry):

$x_1$ $->$

$x_2$ $->$ O $->$ $Y$

$x_3$ $->$

$x_4$ $->$

The weight are:

$C*(\theta_9\theta_1+\theta_{10}\theta_3)$

$C*(\theta_9\theta_2+\theta_{10}\theta_4)$

$C*(\theta_{11}\theta_5+\theta_{12}\theta_7)$

$C*(\theta_{11}\theta_6+\theta_{12}\theta_8)$

Activation function:

$F(\theta,x) = C*\sum\theta x$, defined as C times the sum product of two vectors, $\theta$ and $x$