I was looking at the original implementation of LeNet-5 and I noticed a disparity in different sources. Wikipedia suggests that the non linearity used is the same sigmoid in each layer, some blog posts use a combination of Tanh and sigmoid while Andrew NG said it used some crude non linearity which no one uses today without naming it. I looked at the original paper but it's like 50 pages long and the diagram does not mention the activation functions used explicitly. I searched a bit and the sigmoid function was there and mentioned in context of activations while the tanh function is taken as a squashing function. I'm not sure if that is the same or different as then it used other terms when referring to the sigmoid ones. Anyone knows what's up with this?


In the original paper there are some clarifying statements:

The four inputs to a unit in S2 are added, then multiplied by a trainable coefficient, and added to a trainable bias. The result is passed through a sigmoidal function. (p.7, col.1)

Here, sigmoidal function is generic.

As in classical neural networks, units in layers up to F6 compute.. This weighted sum .. is then passed through a sigmoid squashing function .. The squashing function is a scaled hyperbolic tangent. (p.8, col.1 - but see also Appendix A for details)

Here, "sigmoid squashing function" is used to indicate a scaled "tanh" (remember that tanh is a rescaled logistic sigmoid function).

Therefore, I think Wikipedia's suggestion to use the same "sigmoidal function" is correct. For the sake of precision, the tanh should be used.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.