Some fundamental questions about Transformer

Question

In the Transformer framework, a token as an input (time = $t$) $y^t$ is given by a sum of the original embedding of the token $x^t$ plus, a position embedding factor $v^t$, i.e., \begin{align} y^t = x^t + v^t. \end{align} Here, each symbol denotes an $R^{d}$ vector, where $d$ is the size of the embedding.

The position embedding factor $v^t$ is fixed in the Transformer framework as \begin{align} v^t_{2i} &= \sin (t/T^{2i/d}), \\ v^t_{2i+1} &= \cos (t/T^{2i/d}). \end{align}

Then, I have three questions:

1. I guess that the original embeddings $x^t$ is usually normalized such that $|x^t_i|\leq 1.0$. Then, the position embedding factor has a comparable size since the amplituide is one. Why the Transformer works well? It is very differnt from my intuition. In my understanding, embeddings of the tokens $x^t$ correspond to the expansion by a basis of "meaning" space, if tokens are words. e.g, "airplane" = "machine 80%" and "air" 10% and ... . Then, the comparable fluctuation by the position embedding factor can mess up completely that expansion. It can induce that "airplane" = "human 80%" and "sea" 15%... . How can we explain the reason it works? If the expansion is not done on the meaning space, I still confuse because the position embedding factors can hide the original embeddings.

2. Why the position embedding factor uses both sine and cosine? I have read that, the existence of a linear transformation between different $t$ contributes to possibility of learning. It seems to be true but, the linear transformation again mixes up the expansion in the embedding space! Explicitly, the position embedding factor at $t + k$ can be expressed as \begin{align} \begin{pmatrix} v^{t + k}_{2i}\\ v^{t + k}_{2i+1} \end{pmatrix} = \begin{pmatrix} \cos(k/T^{2i/d}) & \sin(k/T^{2i/d}) \\ -\sin(k/T^{2i/d}) & \cos(k/T^{2i/d}) \end{pmatrix} \begin{pmatrix} v^{t}_{2i}\\ v^{t}_{2i+1} \end{pmatrix}. \end{align} This can be understood as a basis transformation. It mixes up the two different directions in the embedding space. For me, it seems a disaster because each direction corresponds to a certain "mean" in the meaning space. Why this mixture is allowed?

3. Of course, the actual input is the sum $y^t$, not $v^t$. In the above discussions, the appealing (which I do not understand yet) linear transformation is defined on $v^t$. Nevertheless, the original paper and review papers of the Transformer claims that it is the attractive feature. Why can we focus only on the position embedding factor?

I am new to ML (major in physics), so I will miss some fundamental concepts, sorry.

---

(Edit)

What I said "comparable" means that $|x_i^t| \sim |v^t_{2i}|$. Hence, if we add up them, the actual input $y^t$ and $x^t$ will be very different.

If there are some other approaches which change the amplitude as $v^t_{2i} = A \sin(t/T^{2i/d})$ with $A \ll 1$, then it matches with my intuition.

Answer 1

1. Why Does Transformer Work Good with Positional Embeddings?

One question often thought about is why adding positional encodings to token embeddings don't mess up their meanings. Let's explore:

1. High-Dimensional Space: Transformers usually have high dimensions, like 768. But there are these mini sentence transformers that operate effectively with just half; 384 dimensions. 768 gives double degrees of freedom and you can package a lot more information there. So, the number of dimensions is maybe depending on the task. It's a balance.

2. Training: During the training, the Transformer model is learning which features in the high-dimensional space are important. It figures out the right weight for both the token and the position information.

3. Relative Importance Through Attention: Tokens and positions both go into the attention mechanisms. This allows the Transformer to adjust how much importance is given to position compared to token, depending on context.

---

2. Why Both Sine and Cosine in Positional Encodings?

So, why use sine and cosine for positional encodings? There are several reasons:

1. Different Frequencies and Fourier Stuff: Using sine and cosine, the model can capture different types of information across dimensions. This is similar to Fourier Transforms, which decompose complicated signals into simpler waves.

2. Linear Transformations: Sine and cosine make it possible to perform linear transformations between different times. Even though this looks like it confuses the embeddings, the Transformer learns to manage this complexity.

3. Orthogonality: Incorporating sine and cosine brings in a kind of orthogonality. This helps the model to separate out different types of information, similar again to Fourier Transforms.

---

3. Why Positional Encoding Gets Much Attention?

The original "Attention Is All You Need" paper and later works focus a lot on positional encoding. This is because older models like RNNs and LSTMs naturally understand position. But Transformer doesn't. Positional encoding fills this gap and makes it understand sequences.

The token embeddings ( x_t ) are important, for sure. But the positional ones ( v_t ) are a new addition that makes the Transformer work effectively with sequences.