Backpropagation came out around 1974 I believe (paper by Werbos). Looking at the paper, there is no mention of the sigmoid activation function.
When did the sigmoid function become so popular in NNs?
Data Science Stack Exchange is a question and answer site for Data science professionals, Machine Learning specialists, and those interested in learning more about the field. It only takes a minute to sign up.Sign up to join this community
I imagine the sigmoid function would have made its way into ANN's "since the beginning" since it is used in computational neuroscience to approximate the observed activation function of biological neurons. See for example this 1972 paper.
Thanks to @nigelhenry for the paper that demonstrates that sigmoid-ish functions were recognized in actual biological neural networks before backpropagation (+1). Given that, I agree that it makes a natural choice for ANNs. But the earliest ANNs used a discontinuous step function. The introduction of backpropagation (and its predecessors?) required a differentiable activation function, and that does seem to have happened fairly early, though I also haven't found it in Werbos's papers.
I don't have a clear answer, but some references:
A good summary of the history:
Another, but doesn't really touch on activations:
"Learning Internal Representations" by Rumelhart, Hinton, and Williams, 1986:
Page 322: Section "The generalized delta rule" discusses backpropagation.
Page 324: They mention "semilinear" activation functions: nondecreasing, differentiable.
Page 328-329: They have the sigmoid function as activation, as a "useful" semilinear activation. They note the derivative (which is easy to compute, though they don't remark on that), and that the derivative is maximum near zero, so that
weights will be changed most for those units that are...in some sense, not yet committed to being either on or off. This feature, we believe, contributes to the stability of the learning of the system.