How does combining two linear perceptrons create non-linear boundaries?

Question

I don't understand the equation that you get from combining the two linear perceptrons is non-linear?

The video starts with two linear perceptrons with the equations: $$e1 = 5x_1 -2x_2 - 8 = 0 \hspace{10ex} e2 = 7x_1 - 3x_2 + 1 = 0 $$

Note: The bias unit signs flip between the written equation and the neural network diagram. I am using the negative sign since it is continued throughout the rest of the video

Then we go on to combine them with respective weights and biases as follows: $$ 7e_1 + 5e_2 -6 = 0$$ When I do the math, I get: $$ 7e_1 + 5e_2 - 6 = 0 $$ $$ 7(5x_1 -2x_2 - 8) + 5(7x_1 - 3x_2 + 1) - 6 = 0$$ $$ 35x_1 - 14x_2 - 56 + 35x_1 -15x_2 +5 -6 = 0$$ $$ 70x_1 - 29x_2 - 57 = 0$$

The resulting equation is very much linear, however, the idea is that that this generates a non-linear equation (and model).

What am I doing incorrectly in the way I am combining the two models?

Answer 1

You are correct that stacking two layers with a linear activation function on top of each other does not do anything that a single layer could not do (i.e. it is still a linear combination of terms).

This changes, once you use other activation functions. Then, once you combine neurons from the previous layer in the next layer e.g. as $w_0 + w_1 f(e_1) +w_2 f(e_2)$ for some suitable activation function $f$ such as ReLU, the sigmoid function etc., and then stack several such layers on top of each other, you can get substantially more complex functional relationships.