2
$\begingroup$

I have a basic question about activation functions. It is told that they are added to the network to introduce non linearity.
However, the neural network itself is non linear. Isn' it? If we see any neuron with say 3 inputs, the corresponding output equation without an activation function would be - bias + x1*w1 + x2*w2 + x3*w3 .
Above equation is an equation of multiple linear regression. A polynomial curve looks like below which is non linear - enter image description here

Also, when a complex network is formed, in which even if multiple linear blocks are added, the resulting graph would again be a polynomial graph.
All in all, when we have such non linearity, then why do we need to add more non-linear stuff by using activation functions?

Improve this question
$\endgroup$
1
  • $\begingroup$ "multiple linear regression" is not a higher degree "polynomial curve", but it's hard to glean from your question where exactly the confusion is. The neuron equation you provide is linear in the inputs. $\endgroup$
    – Ben Reiniger ♦
    Oct 23, 2022 at 13:42

1 Answer 1

Reset to default
1
$\begingroup$

The neural network is nonlinear if, and only if, you include a nonlinear activation function somewhere in the network.

If you don’t, you wind up with multiples of parameters and biases that technically are nonlinear combinations of one another, but it can be sorted out to give a linear function. Verifying this might be a worthwhile exercise. Perhaps try to do so with a small network like I give here, and take the activation function $A$ to be the identity function. (If, from my equation, you can draw out the network architecture, that’s a good sign!)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.