I have a basic question about activation functions. It is told that they are added to the network to introduce non linearity.
However, the neural network itself is non linear. Isn' it? If we see any neuron with say 3 inputs, the corresponding output equation without an activation function would be -
bias + x1*w1 + x2*w2 + x3*w3 .
Above equation is an equation of multiple linear regression. A polynomial curve looks like below which is non linear -
Also, when a complex network is formed, in which even if multiple linear blocks are added, the resulting graph would again be a polynomial graph.
All in all, when we have such non linearity, then why do we need to add more non-linear stuff by using activation functions?