What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$GELU(x) = xP(X ≤ x) = xΦ(x).$$ which in turn is approximated to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^3)])$$

Could you simplify the equation and explain how it has been approximated.

2 Answers

GELU function

We can expand the cumulative distribution of $$\mathcal{N}(0, 1)$$, i.e. $$\Phi(x)$$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$

Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a stochastic analogy, however mathematically, this is just a definition.

Here is the plot of GELU:

Tanh approximation

For these type of numerical approximations, the key idea is to find a similar function (primarily based on experience), parameterize it, and then fit it to a set of points from the original function.

Knowing that $$\text{erf}(x)$$ is very close to $$\text{tanh}(x)$$

and first derivative of $$\text{erf}(\frac{x}{\sqrt{2}})$$ coincides with that of $$\text{tanh}(\sqrt{\frac{2}{\pi}}x)$$ at $$x=0$$, which is $$\sqrt{\frac{2}{\pi}}$$, we proceed to fit $$\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+ax^2+bx^3+cx^4+dx^5)\right)$$ (or with more terms) to a set of points $$\left(x_i, \text{erf}\left(\frac{x_i}{\sqrt{2}}\right)\right)$$.

I have fitted this function to 20 samples between $$(-1.5, 1.5)$$ (using this site), and here are the coefficients:

By setting $$a=c=d=0$$, $$b$$ was estimated to be $$0.04495641$$. With more samples from a wider range (that site only allowed 20), coefficient $$b$$ will be closer to paper's $$0.044715$$. Finally we get

$$\text{GELU}(x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)\simeq 0.5x\left(1+\text{tanh}\left(\sqrt{\frac{2}{\pi}}(x+0.044715x^3)\right)\right)$$

with mean squared error $$\sim 10^{-8}$$ for $$x \in [-10, 10]$$.

Note that if we did not utilize the relationship between the first derivatives, term $$\sqrt{\frac{2}{\pi}}$$ would have been included in the parameters as follows $$0.5x\left(1+\text{tanh}\left(0.797885x+0.035677x^3\right)\right)$$ which is less beautiful (less analytical, more numerical)!

Utilizing the parity

As suggested by @BookYourLuck, we can utilize the parity of functions to restrict the space of polynomials in which we search. That is, since $$\text{erf}$$ is an odd function, i.e. $$f(-x)=-f(x)$$, and $$\text{tanh}$$ is also an odd function, polynomial function $$\text{pol}(x)$$ inside $$\text{tanh}$$ should also be odd (should only have odd powers of $$x$$) to have $$\text{erf}(-x)\simeq\text{tanh}(\text{pol}(-x))=\text{tanh}(-\text{pol}(x))=-\text{tanh}(\text{pol}(x))\simeq-\text{erf}(x)$$

Previously, we were fortunate to end up with (almost) zero coefficients for even powers $$x^2$$ and $$x^4$$, however in general, this might lead to low quality approximations that, for example, have a term like $$0.23x^2$$ that is being cancelled out by extra terms (even or odd) instead of simply opting for $$0x^2$$.

Sigmoid approximation

A similar relationship holds between $$\text{erf}(x)$$ and $$2\left(\sigma(x)-\frac{1}{2}\right)$$ (sigmoid), which is proposed in the paper as another approximation, with mean squared error $$\sim 10^{-4}$$ for $$x \in [-10, 10]$$.

Here is a Python code for generating data points, fitting the functions, and calculating the mean squared errors:

import math
import numpy as np
import scipy.optimize as optimize

def tahn(xs, a):
return [math.tanh(math.sqrt(2 / math.pi) * (x + a * x**3)) for x in xs]

def sigmoid(xs, a):
return [2 * (1 / (1 + math.exp(-a * x)) - 0.5) for x in xs]

print_points = 0
np.random.seed(123)
# xs = [-2, -1, -.9, -.7, 0.6, -.5, -.4, -.3, -0.2, -.1, 0,
#       .1, 0.2, .3, .4, .5, 0.6, .7, .9, 2]
# xs = np.concatenate((np.arange(-1, 1, 0.2), np.arange(-4, 4, 0.8)))
# xs = np.concatenate((np.arange(-2, 2, 0.5), np.arange(-8, 8, 1.6)))
xs = np.arange(-10, 10, 0.001)
erfs = np.array([math.erf(x/math.sqrt(2)) for x in xs])
ys = np.array([0.5 * x * (1 + math.erf(x/math.sqrt(2))) for x in xs])

# Fit tanh and sigmoid curves to erf points
tanh_popt, _ = optimize.curve_fit(tahn, xs, erfs)
print('Tanh fit: a=%5.5f' % tuple(tanh_popt))

sig_popt, _ = optimize.curve_fit(sigmoid, xs, erfs)
print('Sigmoid fit: a=%5.5f' % tuple(sig_popt))

# curves used in https://mycurvefit.com:
# 1. sinh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))/cosh(sqrt(2/3.141593)*(x+a*x^2+b*x^3+c*x^4+d*x^5))
# 2. sinh(sqrt(2/3.141593)*(x+b*x^3))/cosh(sqrt(2/3.141593)*(x+b*x^3))
y_paper_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + 0.044715 * x**3))) for x in xs])
tanh_error_paper = (np.square(ys - y_paper_tanh)).mean()
y_alt_tanh = np.array([0.5 * x * (1 + math.tanh(math.sqrt(2/math.pi)*(x + tanh_popt[0] * x**3))) for x in xs])
tanh_error_alt = (np.square(ys - y_alt_tanh)).mean()

# curve used in https://mycurvefit.com:
# 1. 2*(1/(1+2.718281828459^(-(a*x))) - 0.5)
y_paper_sigmoid = np.array([x * (1 / (1 + math.exp(-1.702 * x))) for x in xs])
sigmoid_error_paper = (np.square(ys - y_paper_sigmoid)).mean()
y_alt_sigmoid = np.array([x * (1 / (1 + math.exp(-sig_popt[0] * x))) for x in xs])
sigmoid_error_alt = (np.square(ys - y_alt_sigmoid)).mean()

print('Paper tanh error:', tanh_error_paper)
print('Alternative tanh error:', tanh_error_alt)
print('Paper sigmoid error:', sigmoid_error_paper)
print('Alternative sigmoid error:', sigmoid_error_alt)

if print_points == 1:
print(len(xs))
for x, erf in zip(xs, erfs):
print(x, erf)


Output:

Tanh fit: a=0.04485
Sigmoid fit: a=1.70099
Paper tanh error: 2.4329173471294176e-08
Alternative tanh error: 2.698034519269613e-08
Paper sigmoid error: 5.6479106346814546e-05
Alternative sigmoid error: 5.704246564663601e-05

• Why is the approximation needed? Couldn't they just use erf function? May 3, 2019 at 16:51
• Approximate version is very optimization friendly since it involves only tanh (and exp function) for which the fast implementations ares already implemented. This is especially useful if you already use optimized gemm APIs (other parts depending on your NN) and the only part to optimize is the nonlinear functions. May 23 at 11:15

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $$\mathrm{erf}$$. We need to show that $$\mathrm{erf}\left(\frac x {\sqrt2}\right) \approx \tanh\left(\sqrt{\frac2\pi} \left(x + a x^3\right)\right)$$ for $$a \approx 0.044715$$.

For large values of $$x$$, both functions are bounded in $$[-1, 1]$$. For small $$x$$, the respective Taylor series read $$\tanh(x) = x - \frac{x^3}{3} + o(x^3)$$ and $$\mathrm{erf}(x) = \frac{2}{\sqrt{\pi}} \left(x - \frac{x^3}{3}\right) + o(x^3).$$ Substituting, we get that $$\tanh\left(\sqrt{\frac2\pi} \left(x + a x^3\right)\right) = \sqrt\frac{2}{\pi} \left(x + \left(a-\frac{2}{3\pi}\right)x^3\right) + o(x^3)$$ and $$\mathrm{erf}\left(\frac x {\sqrt2}\right) = \sqrt\frac2\pi \left(x - \frac{x^3}{6}\right) + o(x^3).$$ Equating coefficient for $$x^3$$, we find $$a \approx 0.04553992412$$ close to the paper's $$0.044715$$.

• So why the paper does not use $0.04553992412$? Any idea? Aug 5, 2021 at 21:22
• You have to remember that the o(x^3) remainders hide an infinite number of small terms, and these terms are different for each of the functions (small-o notation). Numerical methods would probably take into account some of what gets lost in the higher-order terms. Aug 11, 2021 at 21:19
• Thanks, that makes sense. I am interested to know if the value is approaching theirs if we expand to the o(x^5). Aug 11, 2021 at 22:39