In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form:

$$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$

They argue that :

  • It is easier to approximate with polynomials
  • It is said that it fit the target that it's second derivative is maximal in 1

I tried to start with a function of the form : $f(x) = a * tanh(b*x)$ and derive the value of $a$ and $b$ to match the aforementionned properties.

Any idea of how those constants are derived ? Under what assumptions ? Does it match its expected properties by construction ?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.