I want to know why we need Activation function in DNN hidden layers. I know a bit, like it will help us in,

  1. Increasing model complexity and introduce non-linearity
  2. Avoiding Gradient Vanishing problem.

Still I'm not able to map its requirement. An illustrative example will be definitely helpful.


Activation function is essential part of hidden layers in DNN. Without it the network won't learn any other than a linear equation as a result and usage of DNN is meant to find more complex patterns than linear. There is a mathematical proof that the network reduces to include one weight instead of a weight matrix when no Activation function is present, thus the importance is obvious and huge.

Activation function sort of makes the DNN to test combinations of hidden layer values and that way it helps the network to converge towards the dataset form when combared to validation data and updated accordingly.

The meaning is same as steering wheel for car: without it the brightest car is useless, it would apply for straight roads only, if any (self driving cars exluded).

That is a simplification of the concept, hope you got something out of it.

Very much more complex and thorough answers include in https://ai.stackexchange.com/questions/5493/what-is-the-purpose-of-an-activation-function-in-neural-networks like given by Fatemeh in comments. There you get more points why and how Activation function is needed and used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.