In "Efficient Backprop" (http://yann.lecun.com/exdb/publis/pdf/lecun-98b.pdf), LeCun and others propose a modified tanh activation function of the form:
$$ f(x) = 1.7159 * tanh(\frac{2}{3}*x) $$
They argue that :
- It is easier to approximate with polynomials
- It is said that it fit the target that it's second derivative is maximal in 1
I tried to start with a function of the form : $f(x) = a * tanh(b*x)$ and derive the value of $a$ and $b$ to match the aforementionned properties.
Any idea of how those constants are derived ? Under what assumptions ? Does it match its expected properties by construction ?
