# Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the structure of it, I still don't understand why $$1/10000$$ was used as the scaling factor for the $$pos$$ of a word. Why was this number chosen?

Therefore, the purpose of the 10000 is probably just to make sure the full cycle is extremely large.
If you plot sin(1/10000*x), you will observe that to complete 1 full cycle, x >50k. This is more than sufficient to encode the words which would probably have <1k words.
Note that according to the original equations, the 1st pair of sinusoids is just sin(pos) and cos(pos). In this case, the positional encoding is arguably not unique. i.e. There may be two positions (note: they are integers) that has the same encoding, though this may only occur if we have a very long sentence such that the positions have exactly the same encoding. Though one may argue that since pi is irrational, it is unlikely any integer positions will have same encoding..