# Numerically stable hyperbolic tangent

The hyperbolic tangent is commonly used as an activation function: $$tanh(x) = \frac{e^x - e^{-x}}{e^x + e^{-x}}$$ Although, it is unclear how this function is implemented to be numerically stable in frameworks like pytorch or numpy. Since $$e^x$$ will explode for any $$x \gtrapprox 1000$$, is the function being approximated in some way? Or are there static checks for big input numbers?

For $$|x|<1$$ you can use a polynomial approximation, like Chebyshev. For $$1<|x|\leq E$$, $$E$$ depending on implementation of the formula, you can use a related formula to the one you described.
For $$|x|>E$$ you can use $$\tanh(x) \approx sign(x).$$
• Thanks, using 64-bit floats I expect $E$ to be equal to the natural logarithm of the max value representation of a 64-bit float? np.log(np.finfo(np.float64).max) – Kevin May 6 at 21:01