2
$\begingroup$

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.

In the original "Attention is All You Need" paper by Vaswani et al., positional encodings are implemented using a mix of sine and cosine functions of different frequencies. I understand that these sinusoidal functions provide a unique encoding for each position and that they allow the model to learn to attend to relative positions in a way that is translation invariant.

While going through this blog.

However, I'm wondering why we couldn't simply use normalized positional encodings instead. For example, we could divide the position of each word in the sequence by the maximum sequence length to get a unique positional encoding for each word that falls within a range from 0 to 1. We could handle sequences shorter than the maximum length with padding.

This approach seems more straightforward and it might make training faster or more stable due to the normalized range of the encodings.

Mathematically, for a given position p in a sequence, and a maximum sequence length L, the normalized positional encoding E would be:

E(p) = p / L

The encodings would be unique for each index position and we can easily map the relation positions since the relationship would be linear.

One of the main arguments against this is that we wont be able to handle different sequence lengths since 4/5 and 16/20 would have same ratio but while training we set a max length and the ratios would be the same since padding is incorporated if the sequence is small so the ratios would be the same in all instances and I think that argument doesn't hold.

Improve this question
$\endgroup$
2
  • $\begingroup$ I @multi-arm-bandint, welcome to the site. This question seems like a research question that needs experimentation to be answered, rather than something "known" by the community. $\endgroup$
    – noe
    Aug 3 at 9:28
  • $\begingroup$ @noe In my opinion, asking why a more "obvious" solution isn't being used is a reasonable question. There's definitely reasons why these kinds of "naive" representations aren't used that get at interesting design decisions behind positional embeddings. $\endgroup$
    – Alexander Wan
    Aug 3 at 13:36

2 Answers 2

Reset to default
2
$\begingroup$

I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly recommend you check out the original.

Although it's difficult to say whether or not your suggested representation would work, I can explain some nice properties of the cos and sine encodings, and why those properties aren't present in your suggested representation.

Implicit relative distance bias

You should keep in mind how such representations are used. Without positional embeddings, an attention head attending to a token directly next to it would be treated the same as that attention head attending to a token 1000 tokens away. So one important property is for relative attention to be encoded during the "attending" process. Importantly, this "attending" process is a dot-product. So, in other words, $K_i^\intercal Q_j$ should tell us something about $|i - j|$.

Dot product of position embeddings from https://kazemnejad.com/blog/transformer_architecture_positional_encoding/

This above plot is from the post I linked above. It shows the dot product between positional embeddings at different absolute positions. Notice how closer positions have a higher magnitude, and that such distances are symmetric (i.e., $i - j$ is represented equivalently as $j - i$).

I'm assuming that you're suggesting adding (or concatenating) a $d$ length vector consisting of all $\frac{p}{L}$. In this case, you'd just have higher dot-products for positions later in the sequence, regardless of the relative position. Although, the network could learn a transformation for such dot-products to be useful (and there's a case to be made for not adding such inductive biases into the network), it would take extra "effort" for the model to learn such a transformation.

No dependence on absolute positions

You can see this in the above plot as well: the magnitude only depends on the relative distance between tokens, and the not the absolute distance of the tokens.

The original papers states "We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$" (my emphasis). Consider your representation as a vector:

$$ PE_{pos} = \begin{bmatrix} \frac{p}{L}\\ \frac{p}{L} \end{bmatrix} $$

We want to find some linear transformation $A \in \mathbb{R}^{2x2}$ such that $A \cdot PE_{pos} = PE_{pos + k}$. Although you can find such a matrix, this matrix would depend on the absolute position $pos$. If you wanted to learn such representation in e.g., the $W_k$ or $W_q$ matrices, which are applied to every position, then you'd like the transformation to only depend on the relative position. With the sin and cosine representation on the other hand, you can find such a transformation (see the blog post for how this is done).

Of course, even if you had such a transformation, the resulting vector would not be much use to us (at least directly) as the dot-products do not encode distance information.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ could u share the proper link to the blog currently its pointing to the image itself. I understand that the above plot shows that it may cause an issue. , I do agree that these points don't necessarily prove that a normalized positional encoding scheme couldn't work, but rather, they explain why the sinusoidal positional encodings might be more effective for the transformer architecture. I guess only , empirical results can provide the measure of effectiveness. $\endgroup$
    – mutli-arm-bandit
    Aug 3 at 14:24
  • $\begingroup$ @mutli-arm-bandit Whoops, my mistake -- fixed. $\endgroup$
    – Alexander Wan
    Aug 3 at 14:27
  • $\begingroup$ so tried creating a simple one for normalised encoding assume we use a 3 sequence W(1) = [0.5 + 0.33] = [0.83] (token + pos) W(2) = [0.6 + 0.67] = [1.27] W(3) = [0.7 + 1.0] = [1.7] Dot(W(1), W(1)) = 0.83 * 0.83 = 0.69 Dot(W(1), W(2)) = 0.83 * 1.27 = 1.05 Dot(W(1), W(3)) = 0.83 * 1.7 = 1.41 Dot(W(2), W(2)) = 1.27 * 1.27 = 1.61 Dot(W(2), W(3)) = 1.27 * 1.7 = 2.16 Dot(W(3), W(3)) = 1.7 * 1.7 = 2.89 [[0.69, 1.05, 1.41], [1.05, 1.61, 2.16], [1.41, 2.16, 2.89]] so yep the attention scores increase we move towards later positions in the sequence,regardless of the rel positions. $\endgroup$
    – mutli-arm-bandit
    Aug 3 at 14:36
2
$\begingroup$

Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409

For me, the takeaways were:

  • The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would extrapolate beyond sentence lengths seen in training. Turns out it doesn't.
  • The relative positional embedding ideas used in T5 and GPT-J work better, but still not that well.
  • It was chosen for and works well in BLOOM
  • I think ALiBi results only apply to a decoder-only transformer; I need to track down if there has been follow-up work for encoder-only and encoder-decoder transformers.

Your simple idea will also work, as it has a different value for each position. I suspect it won't extrapolate well, and I suspect it will be inferior to the more established ways, mainly as it is an obvious approach, implying people have tried it. (I'm sure I've seen it mentioned in a paper before, but couldn't tell you which one, sorry.)

Improve this answer
$\endgroup$
1
  • $\begingroup$ I think the main idea from the Attention is all you need paper , is that the sin and cos method dosent have a limit on the context length , this is important in the decoder stage since we have no idea how long the answer would be , but paper such as BERT use learned positional embedding which use a fixed length so that removed the need for sin and cos , so maybe they did try with what i suggested and it didnt work so I do agree with u and thanks for the great paper reference. $\endgroup$
    – mutli-arm-bandit
    Aug 28 at 20:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.