3
$\begingroup$

Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.

When did the sigmoid function become so popular in NNs?

$\endgroup$
3
$\begingroup$

I imagine the sigmoid function would have made its way into ANN's "since the beginning" since it is used in computational neuroscience to approximate the observed activation function of biological neurons. See for example this 1972 paper.

| improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ At first skim, this doesn't seem to be about ANNs the way we now think of them, but instead about more direct biological models of neuron interactions? Of course, that may be part of the reason for the adoption of sigmoid-ish activation functions in ANNs. $\endgroup$ – Ben Reiniger Feb 28 at 2:43
  • $\begingroup$ Gotcha, I always thought they used a simply step function for neuroscience, did not know they always used a sigmoid! Thanks for this. $\endgroup$ – Shinobii Feb 28 at 3:05
  • $\begingroup$ @BenReiniger You're right, this is a computational neuroscience paper, not ANN. I will edit my answer $\endgroup$ – nigelhenry Feb 28 at 6:03
2
$\begingroup$

Thanks to @nigelhenry for the paper that demonstrates that sigmoid-ish functions were recognized in actual biological neural networks before backpropagation (+1). Given that, I agree that it makes a natural choice for ANNs. But the earliest ANNs used a discontinuous step function. The introduction of backpropagation (and its predecessors?) required a differentiable activation function, and that does seem to have happened fairly early, though I also haven't found it in Werbos's papers.

I don't have a clear answer, but some references:

A good summary of the history:
http://www.andreykurenkov.com/writing/ai/a-brief-history-of-neural-nets-and-deep-learning/

Another, but doesn't really touch on activations:
http://people.idsia.ch/~juergen/who-invented-backpropagation.html

"Learning Internal Representations" by Rumelhart, Hinton, and Williams, 1986:
Page 322: Section "The generalized delta rule" discusses backpropagation.
Page 324: They mention "semilinear" activation functions: nondecreasing, differentiable.
Page 328-329: They have the sigmoid function as activation, as a "useful" semilinear activation. They note the derivative (which is easy to compute, though they don't remark on that), and that the derivative is maximum near zero, so that

weights will be changed most for those units that are...in some sense, not yet committed to being either on or off. This feature, we believe, contributes to the stability of the learning of the system.

| improve this answer | |
$\endgroup$
  • $\begingroup$ Right, I know that the first NNs used logical (binary) neurons with a threshold (step function). Just curious if backpropagation came AFTER a differential function was suggested (sigmoid) or if they went hand-in-hand etc. Thanks for the great response! $\endgroup$ – Shinobii Feb 28 at 17:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.