# How do multiple linear neurons together allow for nonlinearity in a neural network?

As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says: "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?

• ReLUs are merely piecewise-linear. – Emre Sep 16 '16 at 4:35
• Exactly! So how would using a ReLu along with other linear operators allow you to access nonlinearity in a neural network? – Monica Heddneck Sep 16 '16 at 4:39
• @Emre already answered you. Piecewise-linear means it's only linear in segments. – HelloWorld Sep 16 '16 at 5:05
• So the statement "To learn non-linear decision boundaries when classifying the output, multiple neurons are required." is incorrect? – Monica Heddneck Sep 16 '16 at 5:41

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).

• Great answer. Would you agree with the answer to this question -- that we can combine ReLus to approximate nonlinear functions? stats.stackexchange.com/questions/141960/… – Monica Heddneck Sep 16 '16 at 7:50
• @MonicaHeddneck The answer there is correct, and is really what I said without the mathematics details (pipe up a dozen of non-linear outputs). – HelloWorld Sep 16 '16 at 7:53
• Wow. That's amazing and I've learned so much. How would you know what type of these more complicated ReLus you'd need...or what order to put them in? – Monica Heddneck Sep 16 '16 at 7:54
• @MonicaHeddneck That's parameter optimisation. Nobody knows for sure, we'll need to train the model and evaluate. ML is hard... – HelloWorld Sep 16 '16 at 7:55