0
$\begingroup$

The hyperbolic tangent is commonly used as an activation function: $$ tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}} $$ Although, it is unclear how this function is implemented to be numerically stable in frameworks like pytorch or numpy. Since $e^x$ will explode for any $x \gtrapprox 1000$, is the function being approximated in some way? Or are there static checks for big input numbers?

$\endgroup$
2
$\begingroup$

Commonly you divide the domain up into different accuracy,stability regimes and apply a different approximation within each.

For $|x|<1$ you can use a polynomial approximation, like Chebyshev. For $1<|x|\leq E$, $E$ depending on implementation of the formula, you can use a related formula to the one you described.

For $|x|>E$ you can use $\tanh(x) \approx sign(x).$

More details here from a documented implementation.

$\endgroup$
1
  • $\begingroup$ Thanks, using 64-bit floats I expect $E$ to be equal to the natural logarithm of the max value representation of a 64-bit float? np.log(np.finfo(np.float64).max) $\endgroup$ – Kevin May 6 at 21:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.