Why does the transformer positional encoding use both sine and cosine?

Question

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.

I am wondering why it needs to use both sine and cosine though instead of just one or the other?

Answer 1

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.

In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you need to know the length of the sequence beforehand.

This is why the authors propose a cyclic solution. Sin and cos functions can benefit from this idea and thus are the choice here. But why using both instead of one?

It is not clearly answered in the original paper, but if you go though this article(which I urge you to) the reason is that you can use a linear transformation to go from the two functions, to the same functions with an offset:

And if you wonder why having this property available is useful, if you dig into the comments below the article you will find the explanation you seek:

I suppose that this encoding framework enables the model to attend to relative positions by simply generating a transformation matrix, which only depends on the k. To be clear, I hypothesize that given any input PE(pos), the model can create the attention query matrix Q that targets PE(pos+k) by multiplying the PE(pos) with a weight matrix T (the transformation matrix). The weight matrix T, which could be parameters of a single feed-forward layer, can be learned during the training process.

Basically the model can learn a matrix not dependant on t as part o the trining to generalize all the range of encoding values.

Can we just use one of the other?

You could encode the values of the position, yes. But you would not have the property of having this linear transformation available, which seems to be a very important part for the learning process to succeed.

Answer 2

There are three main reasons for using cosine and sine functions in the transformer encoding formula:

1. Periodicity: They have repetitive patterns, which is useful for attention mechanism in a trackable manner.
2. Constrained values: It bound the values between -1 and +1. In the actual attention matrix, words far away will exhibit lesser values.
3. Easy to extrapolate for long sequences and easy to compute.