In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:

$ PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/d_{model}} ) $

$ PE(pos, 2dim+1) = cos(pos / 10000 ^ {2dim/d_{model}} ) $

The text states that "for any fixed offset $k$, $PE(pos+k)$ can be represented as a linear function of $PE(pos)$". This did not seem obvious due to me due to the nonlinearity of the sine function. Other resources like Attention is all you need Explained mention this property but do not go deeper into it.

I decided to experiment with this by attempting to map a number of outputs from the $PE(pos)$ function to outputs with a given offset $k$.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def PE_even(pos, dim):
    """ This corresponds to the 2dim state. """
    size = 10000
    d_Model = 512
    return np.sin(pos / (size ** (2 * dim / d_Model)))

positions = pd.Series(np.arange(0, 1000))

k = 10
a = positions.apply(lambda p: PE_even(p, 64))
b = positions.apply(lambda p: PE_even(p + k, 64))

X = a.values.reshape(-1, 1)
y = b.values
r = LinearRegression()
r.fit(X, y)
r.score(X, y)

However, no offset nor any range of numbers in these dimensions yields a suitable fit. As $k$ increases and the sine waves resulting from the $PE(pos)$ function get out of sync, the correlation of the transformation and the truth decreases. Even using simple neural networks with and without linearities does not yield a good fit.

Did I misapprehend the statement in the paper, or is my code or understanding of the underlying math here faulty?


1 Answer 1


I elected to ask this question on the Mathematics Stack Exchange and I thought it prudent to add the answer here:


From what I have learned from @Servaes, who was kind enough to answer the question, there is a function


that allows for the property of transforming any positional encoding into one with a given offset. However, due to the required use of the sin() and cos() functions this is not a linear transformation.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.