In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^t + v^t. \end{align} Here, each symbol denotes an $R^{d}$ vector, where $d$ is the size of the embedding.
The position embedding factor $v^t$ is fixed in the Transformer framework as \begin{align} v^t_{2i} &= \sin (t/T^{2i/d}), \\ v^t_{2i+1} &= \cos (t/T^{2i/d}). \end{align}
Then, I have three questions:
I guess that the original embeddings $x^t$ is usually normalized such that $|x^t_i|\leq 1.0$. Then, the position embedding factor has a comparable size since the amplituide is one. Why the Transformer works well? It is very differnt from my intuition. In my understanding, embeddings of the tokens $x^t$ correspond to the expansion by a basis of "meaning" space, if tokens are words. e.g, "airplane" = "machine 80%" and "air" 10% and ... . Then, the comparable fluctuation by the position embedding factor can mess up completely that expansion. It can induce that "airplane" = "human 80%" and "sea" 15%... . How can we explain the reason it works? If the expansion is not done on the meaning space, I still confuse because the position embedding factors can hide the original embeddings.
Why the position embedding factor uses both sine and cosine? I have read that, the existence of a linear transformation between different $t$ contributes to possibility of learning. It seems to be true but, the linear transformation again mixes up the expansion in the embedding space! Explicitly, the position embedding factor at $t + k$ can be expressed as \begin{align} \begin{pmatrix} v^{t + k}_{2i}\\ v^{t + k}_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(k/T^{2i/d}) & \sin(k/T^{2i/d}) \\ -\sin(k/T^{2i/d}) & \cos(k/T^{2i/d}) \end{pmatrix} \begin{pmatrix} v^{t}_{2i}\\ v^{t}_{2i+1} \end{pmatrix}. \end{align} This can be understood as a basis transformation. It mixes up the two different directions in the embedding space. For me, it seems a disaster because each direction corresponds to a certain "mean" in the meaning space. Why this mixture is allowed?
Of course, the actual input is the sum $y^t$, not $v^t$. In the above discussions, the appealing (which I do not understand yet) linear transformation is defined on $v^t$. Nevertheless, the original paper and review papers of the Transformer claims that it is the attractive feature. Why can we focus only on the position embedding factor?
I am new to ML (major in physics), so I will miss some fundamental concepts, sorry.
(Edit)
What I said "comparable" means that $|x_i^t| \sim |v^t_{2i}|$. Hence, if we add up them, the actual input $y^t$ and $x^t$ will be very different.
If there are some other approaches which change the amplitude as $v^t_{2i} = A \sin(t/T^{2i/d})$ with $A \ll 1$, then it matches with my intuition.