How do multiple linear neurons together allow for nonlinearity in a neural network?

Question

As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.

For example, this answer says: "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:

We have a multiply, add, ReLu, multiply and add, all in series.

From studying math, I know that a composite function made out of linear functions is itself linear.

So how do you coax non-linearity out of multiple linear functions?

Answer 1

The phase

"To learn non-linear decision boundaries when classifying the output, multiple neurons are required."

is NOT correct. More precisely, it should be:

"To learn non-linear decision boundaries when classifying the output, we need a non-linear activation function."

To understand why, imagine you have a network with many layers and nodes (multiple neurons in your question). If you don't have a non-linear activation function such as ReLu or sigmoid, your network is just a linear combination of bias and weights. Your network won't be useful for classifying non-linear decision boundary. But if your inputs can be linearly separable, you don't need neutral network...

That's why all neutral networks almost always have a non-linear activation function. ReLu is the most popular, but there are other possibilites. When you pipe up a dozen of non-linear outputs like in neutral network, your network will be able to classify a non-linear decision boundary. The more your have, the better it can perform (but also easier for overfitting).