# Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.

I am wondering why it needs to use both sine and cosine though instead of just one or the other?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.

In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you need to know the length of the sequence beforehand.

This is why the authors propose a cyclic solution. Sin and cos functions can benefit from this idea and thus are the choice here. But why using both instead of one?

It is not clearly answered in the original paper, but if you go though this article(which I urge you to) the reason is that you can use a linear transformation to go from the two functions, to the same functions with an offset: And if you wonder why having this property available is useful, if you dig into the comments below the article you will find the explanation you seek:

I suppose that this encoding framework enables the model to attend to relative positions by simply generating a transformation matrix, which only depends on the k. To be clear, I hypothesize that given any input PE(pos), the model can create the attention query matrix Q that targets PE(pos+k) by multiplying the PE(pos) with a weight matrix T (the transformation matrix). The weight matrix T, which could be parameters of a single feed-forward layer, can be learned during the training process.

Basically the model can learn a matrix not dependant on t as part o the trining to generalize all the range of encoding values.

## Can we just use one of the other?

You could encode the values of the position, yes. But you would not have the property of having this linear transformation available, which seems to be a very important part for the learning process to succeed.

• What do you mean by the 'lack of boundary in the value and the magnitude'? Dec 11, 2021 at 20:39
• @Yamaneko I believe this means that if it is sequential it could go to infinity (you need to know the dimension to know how high it goes), but if it is using this approach it is always between [-1 and 1] - so no matter how large the input is you know the boundaries of what the encoding can be. Jan 8 at 2:05

There are three main reasons for using cosine and sine functions in the transformer encoding formula:

1. Periodicity: They have repetitive patterns, which is useful for attention mechanism in a trackable manner.
2. Constrained values: It bound the values between -1 and +1. In the actual attention matrix, words far away will exhibit lesser values.
3. Easy to extrapolate for long sequences and easy to compute.