I'm working with Transformer models for sequence-to-sequence tasks and I'm trying to fully understand the use of positional encodings in these models.
In the original "Attention is All You Need" paper by Vaswani et al., positional encodings are implemented using a mix of sine and cosine functions of different frequencies. I understand that these sinusoidal functions provide a unique encoding for each position and that they allow the model to learn to attend to relative positions in a way that is translation invariant.
While going through this blog.
However, I'm wondering why we couldn't simply use normalized positional encodings instead. For example, we could divide the position of each word in the sequence by the maximum sequence length to get a unique positional encoding for each word that falls within a range from 0 to 1. We could handle sequences shorter than the maximum length with padding.
This approach seems more straightforward and it might make training faster or more stable due to the normalized range of the encodings.
Mathematically, for a given position p in a sequence, and a maximum sequence length L, the normalized positional encoding E would be:
E(p) = p / L
The encodings would be unique for each index position and we can easily map the relation positions since the relationship would be linear.
One of the main arguments against this is that we wont be able to handle different sequence lengths since 4/5 and 16/20 would have same ratio but while training we set a max length and the ratios would be the same since padding is incorporated if the sequence is small so the ratios would be the same in all instances and I think that argument doesn't hold.