Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

Question

I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.

In the original "Attention is All You Need" paper by Vaswani et al., positional encodings are implemented using a mix of sine and cosine functions of different frequencies. I understand that these sinusoidal functions provide a unique encoding for each position and that they allow the model to learn to attend to relative positions in a way that is translation invariant.

While going through this blog.

However, I'm wondering why we couldn't simply use normalized positional encodings instead. For example, we could divide the position of each word in the sequence by the maximum sequence length to get a unique positional encoding for each word that falls within a range from 0 to 1. We could handle sequences shorter than the maximum length with padding.

This approach seems more straightforward and it might make training faster or more stable due to the normalized range of the encodings.

Mathematically, for a given position p in a sequence, and a maximum sequence length L, the normalized positional encoding E would be:

E(p) = p / L

The encodings would be unique for each index position and we can easily map the relation positions since the relationship would be linear.

One of the main arguments against this is that we wont be able to handle different sequence lengths since 4/5 and 16/20 would have same ratio but while training we set a max length and the ratios would be the same since padding is incorporated if the sequence is small so the ratios would be the same in all instances and I think that argument doesn't hold.

Answer 1

I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly recommend you check out the original.

Although it's difficult to say whether or not your suggested representation would work, I can explain some nice properties of the cos and sine encodings, and why those properties aren't present in your suggested representation.

Implicit relative distance bias

You should keep in mind how such representations are used. Without positional embeddings, an attention head attending to a token directly next to it would be treated the same as that attention head attending to a token 1000 tokens away. So one important property is for relative attention to be encoded during the "attending" process. Importantly, this "attending" process is a dot-product. So, in other words, $K_i^\intercal Q_j$ should tell us something about $|i - j|$.

This above plot is from the post I linked above. It shows the dot product between positional embeddings at different absolute positions. Notice how closer positions have a higher magnitude, and that such distances are symmetric (i.e., $i - j$ is represented equivalently as $j - i$).

I'm assuming that you're suggesting adding (or concatenating) a $d$ length vector consisting of all $\frac{p}{L}$. In this case, you'd just have higher dot-products for positions later in the sequence, regardless of the relative position. Although, the network could learn a transformation for such dot-products to be useful (and there's a case to be made for not adding such inductive biases into the network), it would take extra "effort" for the model to learn such a transformation.

No dependence on absolute positions

You can see this in the above plot as well: the magnitude only depends on the relative distance between tokens, and the not the absolute distance of the tokens.

The original papers states "We chose this function because we hypothesized it would allow the model to easily learn to attend by relative positions, since for any fixed offset $k$, $PE_{pos+k}$ can be represented as a linear function of $PE_{pos}$" (my emphasis). Consider your representation as a vector:

$$ PE_{pos} = \begin{bmatrix} \frac{p}{L}\\ \frac{p}{L} \end{bmatrix} $$

We want to find some linear transformation $A \in \mathbb{R}^{2x2}$ such that $A \cdot PE_{pos} = PE_{pos + k}$. Although you can find such a matrix, this matrix would depend on the absolute position $pos$. If you wanted to learn such representation in e.g., the $W_k$ or $W_q$ matrices, which are applied to every position, then you'd like the transformation to only depend on the relative position. With the sin and cosine representation on the other hand, you can find such a transformation (see the blog post for how this is done).

Of course, even if you had such a transformation, the resulting vector would not be much use to us (at least directly) as the dot-products do not encode distance information.

Answer 2

Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409

For me, the takeaways were:

• The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would extrapolate beyond sentence lengths seen in training. Turns out it doesn't.
• The relative positional embedding ideas used in T5 and GPT-J work better, but still not that well.
• It was chosen for and works well in BLOOM
• I think ALiBi results only apply to a decoder-only transformer; I need to track down if there has been follow-up work for encoder-only and encoder-decoder transformers.

Your simple idea will also work, as it has a different value for each position. I suspect it won't extrapolate well, and I suspect it will be inferior to the more established ways, mainly as it is an obvious approach, implying people have tried it. (I'm sure I've seen it mentioned in a paper before, but couldn't tell you which one, sorry.)