# How can positional encodings including a sine operation be linearly transformable for any offset?

In the paper "Attention is all you need" the authors add a positional encoding to each token in the sequence (section 3.5). The following encoding is chosen:

$$PE(pos, 2dim) = sin(pos / 10000 ^ {2dim/d_{model}} )$$

$$PE(pos, 2dim+1) = cos(pos / 10000 ^ {2dim/d_{model}} )$$

The text states that "for any fixed offset $$k$$, $$PE(pos+k)$$ can be represented as a linear function of $$PE(pos)$$". This did not seem obvious due to me due to the nonlinearity of the sine function. Other resources like Attention is all you need Explained mention this property but do not go deeper into it.

I decided to experiment with this by attempting to map a number of outputs from the $$PE(pos)$$ function to outputs with a given offset $$k$$.

import numpy as np
import pandas as pd
from sklearn.linear_model import LinearRegression

def PE_even(pos, dim):
""" This corresponds to the 2dim state. """
size = 10000
d_Model = 512
return np.sin(pos / (size ** (2 * dim / d_Model)))

positions = pd.Series(np.arange(0, 1000))

k = 10
a = positions.apply(lambda p: PE_even(p, 64))
b = positions.apply(lambda p: PE_even(p + k, 64))

X = a.values.reshape(-1, 1)
y = b.values
r = LinearRegression()
r.fit(X, y)
r.score(X, y)


However, no offset nor any range of numbers in these dimensions yields a suitable fit. As $$k$$ increases and the sine waves resulting from the $$PE(pos)$$ function get out of sync, the correlation of the transformation and the truth decreases. Even using simple neural networks with and without linearities does not yield a good fit.

Did I misapprehend the statement in the paper, or is my code or understanding of the underlying math here faulty?

$$\operatorname{PE}(\text{pos}+k,2d)=\operatorname{PE}(\text{pos},2d)\cos(k/c^d)+\operatorname{PE}(\text{pos},2d+1)\sin(k/c^d)$$