1
$\begingroup$

I don't understand the equation that you get from combining the two linear perceptrons is non-linear?

The video starts with two linear perceptrons with the equations: $$e1 = 5x_1 -2x_2 - 8 = 0 \hspace{10ex} e2 = 7x_1 - 3x_2 + 1 = 0 $$

Note: The bias unit signs flip between the written equation and the neural network diagram. I am using the negative sign since it is continued throughout the rest of the video

Then we go on to combine them with respective weights and biases as follows: $$ 7e_1 + 5e_2 -6 = 0$$ When I do the math, I get: $$ 7e_1 + 5e_2 - 6 = 0 $$ $$ 7(5x_1 -2x_2 - 8) + 5(7x_1 - 3x_2 + 1) - 6 = 0$$ $$ 35x_1 - 14x_2 - 56 + 35x_1 -15x_2 +5 -6 = 0$$ $$ 70x_1 - 29x_2 - 57 = 0$$

The resulting equation is very much linear, however, the idea is that that this generates a non-linear equation (and model).

What am I doing incorrectly in the way I am combining the two models?

$\endgroup$
5
$\begingroup$

You are correct that stacking two layers with a linear activation function on top of each other does not do anything that a single layer could not do (i.e. it is still a linear combination of terms).

This changes, once you use other activation functions. Then, once you combine neurons from the previous layer in the next layer e.g. as $w_0 + w_1 f(e_1) +w_2 f(e_2)$ for some suitable activation function $f$ such as ReLU, the sigmoid function etc., and then stack several such layers on top of each other, you can get substantially more complex functional relationships.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.