I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the structure of it, I still don't understand why $1/10000$ was used as the scaling factor for the $pos$ of a word. Why was this number chosen?
This is my understanding, feel free to correct me, I feel looking at how
n visually impact the positional encoding matrix helpful.
Here is the same positional encoding (sequence length=100 and dimension=512) but with different values of
If you think about the purpose of positional encoding: providing a unique vector representation encoding the position of each token, it means each row represent a position and the further apart two positions the greater the distance between this vector.
Now, look what happens when you compute the cosine distance between the first vector and each remaining one:
orange: n = 10000 blue: n = 20 green: n = 1
With large n the distance is a monotonically increasing function of the position (in term of index in the sequence) which make sense since we're talking about position of token. But it's not the case with smaller value of n.
Now I don't know if there's any rule regarding how to set an appropriate
n depending on the other parameters.
I offer a slightly different perspective. Your typical sentence is 15-20 words long. Their shortest wavelength is ~6. Their longest wavelength is ~60000. As you can see from Yoan's first fig, for sentences up to length 20, the positional encoding for about half the channels (i>256) are basically constant because the wavelength is so long. Thus half the channels get no positional information, which may actually be good for some information. After all, adding (as opposed to concatenating) the positional encoding is a little strange in that it could corrupt useful information in the input.
Amirhossein's blog post explains the intuition for positional encoding very well.
My takeaway from the blog is that: Consider just a pair of sinusoids (sine and cosine). Suppose we are within 1 full cycle (e.g. 0 to 2pi), the resulting encoding is basically guaranteed to be unique. I.e. there is a 1-to-1 mapping from real numbers (1, 1.5, 2, 2.34,etc.), x, to the pair of (sin(x), cos(x)). Thus, encoded to this 2-tuple vector.
Therefore, the purpose of the
10000 is probably just to make sure the full cycle is extremely large.
If you plot
sin(1/10000*x), you will observe that to complete 1 full cycle, x >50k. This is more than sufficient to encode the words which would probably have <1k words.
Note that according to the original equations, the 1st pair of sinusoids is just
cos(pos). In this case, the positional encoding is arguably not unique. i.e. There may be two positions (note: they are integers) that has the same encoding, though this may only occur if we have a very long sentence such that the positions have exactly the same encoding. Though one may argue that since
pi is irrational, it is unlikely any integer positions will have same encoding..