2
$\begingroup$

I was reading this article where I came across the following statement in the context of "Why do we use sigmoid activation function in Neural Nets?":

The assumption of a dependent variable to follow a sigmoid function inherently assumes a Gaussian distribution for the independent variable which is a general distribution we see for a lot of randomly occurring events and this is a good generic distribution to start with.

Could someone elaborate on this relationship between the two?

$\endgroup$

1 Answer 1

0
$\begingroup$

Yes sure, the core idea is that the sigmoid function approximate very nicely the cumulative density function of a Gaussian distribution. Therefore, if the input of a neuron is a continuous Gaussian distribution, the probability of the output being inferior to a certain value will follow (almost) the shape of the sigmoid function.

A detailed explanation can be found p16 of the following article:

https://www.researchgate.net/publication/220233479_Catastrophic_Forgetting_and_the_Pseudorehearsal_Solution_in_Hopfield-type_Networks

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.