I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in,

  1. Increasing model complexity and introduce non-linearity
  2. Avoiding Gradient Vanishing problem.

Still I'm not able to map its requirement. An illustrative example will be definitely helpful.


Activation function is an essential part of hidden layers in DNN. Without it the network won't learn any other than a linear equation as a result and usage of DNN is meant to find more complex patterns than linear. There is a mathematical proof that the network reduces to include one weight instead of a weight matrix when no Activation function is present, thus the importance is obvious and huge.

Activation function sort of makes the DNN to test combinations of hidden layer values and that way it helps the network to converge towards the dataset form when compared to validation data and updated accordingly.

The meaning is the same as a steering wheel for a car: without it the brightest car is useless, it would apply for straight roads only if any (self-driving cars excluded).

That is a simplification of the concept, hope you got something out of it.

Very much more complex and thorough answers included in What is the purpose of an activation function in neural networks? like given by Fatemeh in comments. There you get more points on why and how the Activation function is needed and used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.