11
$\begingroup$

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.

I am wondering why it needs to use both sine and cosine though instead of just one or the other?

$\endgroup$

2 Answers 2

10
$\begingroup$

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.

In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you need to know the length of the sequence beforehand.

This is why the authors propose a cyclic solution. Sin and cos functions can benefit from this idea and thus are the choice here. But why using both instead of one?

It is not clearly answered in the original paper, but if you go though this article(which I urge you to) the reason is that you can use a linear transformation to go from the two functions, to the same functions with an offset:

enter image description here

And if you wonder why having this property available is useful, if you dig into the comments below the article you will find the explanation you seek:

I suppose that this encoding framework enables the model to attend to relative positions by simply generating a transformation matrix, which only depends on the k. To be clear, I hypothesize that given any input PE(pos), the model can create the attention query matrix Q that targets PE(pos+k) by multiplying the PE(pos) with a weight matrix T (the transformation matrix). The weight matrix T, which could be parameters of a single feed-forward layer, can be learned during the training process.

Basically the model can learn a matrix not dependant on t as part o the trining to generalize all the range of encoding values.

Can we just use one of the other?

You could encode the values of the position, yes. But you would not have the property of having this linear transformation available, which seems to be a very important part for the learning process to succeed.

$\endgroup$
2
  • $\begingroup$ What do you mean by the 'lack of boundary in the value and the magnitude'? $\endgroup$
    – Yamaneko
    Commented Dec 11, 2021 at 20:39
  • 1
    $\begingroup$ @Yamaneko I believe this means that if it is sequential it could go to infinity (you need to know the dimension to know how high it goes), but if it is using this approach it is always between [-1 and 1] - so no matter how large the input is you know the boundaries of what the encoding can be. $\endgroup$ Commented Jan 8, 2023 at 2:05
0
$\begingroup$

There are three main reasons for using cosine and sine functions in the transformer encoding formula:

  1. Periodicity: They have repetitive patterns, which is useful for attention mechanism in a trackable manner.
  2. Constrained values: It bound the values between -1 and +1. In the actual attention matrix, words far away will exhibit lesser values.
  3. Easy to extrapolate for long sequences and easy to compute.
$\endgroup$
1
  • $\begingroup$ You give an answer for using cosine OR sinus functions. The question asks for the reason for using cosine AND sinus functions. $\endgroup$
    – PascalIv
    Commented Oct 24, 2023 at 11:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.