As I understand it, the point of architecting multiple layers in a neural network is so that you can have non-linearity represented in your deep network.
For example, this answer says: "To learn non-linear decision boundaries when classifying the output, multiple neurons are required."
When I watch online tutorials and whatnot, I see networks described as in the screenshot below. In cases like this, I see a series of linear classifiers:
We have a multiply, add, ReLu, multiply and add, all in series.
From studying math, I know that a composite function made out of linear functions is itself linear.
So how do you coax non-linearity out of multiple linear functions?