Positional encoding using sine-cosine functions is often used in transformer models.

Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the embedding size. This positional encoding layer encodes $X$’s position $P \in R^{l\times d}$ and outputs $P + X$

The position $P$ is a 2-D matrix, where $i$ refers to the order in the sentence, and $j$ refers to the position along the embedding vector dimension. In this way, each value in the origin sequence is then maintained using the equations below:

$${P_{i, 2j} = \sin \bigg( \frac{i}{10000^{2j/d}}} \bigg) $$

$${P_{i, 2j+1} = \cos \bigg( \frac{i}{10000^{2j/d}}} \bigg)$$

for $i = 0,..., l-1$ and $j=0,...[(d-1)/2]$

I understand the transormation across the time dimension $i$ but why do we need the transformation across the embedding size dimension $j$? Since we are adding the position, wouldn't sin-cos just on time dimension be sufficient to encode the position?


Answer 1 - Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.

I still don't understand how the embedding dimensionality will be reduced to 1 if the same positional vector is added. Say we have an input $X$ of zeros with 4 dimensions - $d_0, d_1, d_2, d_3$ and 3 time steps - $t_0, t_1, t_2$

$$ \begin{matrix} & d_0 & d_1 & d_2 & d_3\\ t_0 & 0 & 0 & 0 & 0\\ t_1 & 0 & 0 & 0 & 0\\ t_2 & 0 & 0 & 0 & 0\\ \end{matrix} $$

If $d_0$ and $d_2$ are the same vectors $[0, 0, 0]$, and the meaning of position i.e time step is the same, why do they need to have different positional vectors? Why can't $d_0$ and $d_2$ be the same after positional encoding if the input $d_0$ and $d_2$ are the same?

As for the embedding dimensionality reducing to 1, I don't see why that would happen. Isn't the embedding dimensionality dependent on the input matrix $X$. If I add constants to it, the dimensionality will not change, no?

I may be missing something more fundamental here and would like to know where am I going wrong.

  • $\begingroup$ wrt embedding dimensionality reducing to 1, I'm referring to the input matrix and not the positional matrix. Yes, for the positional matrix it would become 1. But the whole question is why do we need it in the first place. $\endgroup$
    – kyc12
    Commented Nov 24, 2020 at 18:40
  • $\begingroup$ I edited my answer with further details based on your edits. $\endgroup$
    – noe
    Commented Dec 2, 2020 at 14:21

1 Answer 1


First, let's reason why positional embeddings are needed at all:

A multi-head attention layer of the Transformer architecture performs computations that are position-independent. This means that, if the same inputs are received at two different positions, the attention heads in the layer would return the same value at the two positions.

Note that this is different from LSTMs and other recurrent architectures which, apart from the input, receive the state from the previous time step.

The role of positional embeddings is to supply information regarding the position of each token. This allows the attention layer to compute results that are context-dependent, that is, two tokens with the same value in the input sentence would get different representations.

Second, let's clarify why having a fixed formula to compute the positional embeddings:

Positional embeddings can be handled as "normal" embedding matrixes and therefore can be trained with the rest of the network. These are "trainable positional embeddings". With this kind of positional embeddings, after each training step, the positional embedding matrix is updated together with the rest of the parameters.

However, we can obtain the same level of performance (translation quality, perplexity, or whatever other measure being used) if, instead of training the positional embeddings, we used the formula proposed in the original transformer paper.

This saves us from having to train a very big embedding matrix.

Now, about why using the "embedding size dimension" in the formula:

We need different values in each position of the embedded vector. Having the same value in each position of the vector would leave us with an "effective" embedding size of 1, as we are wasting the other $d-1$ positions.

In order to compute different values for each position of the embedded vector, we need an independent variable that we use to compute the value at each position based on it. We don't have any other suitable variable but the position itself. That's why it is used in the formula.

Old answer:

Making the embedding vector independent from the "embedding size dimension" would lead to having the same value in all positions, and this would reduce the effective embedding dimensionality to 1.

The formula uses the embedding size dimension to be able to provide different values within each embedded vector.

  • $\begingroup$ But we are adding the position matrix P to the input X. A given row (time step) in P will have same values but X will not. The result would be that each row has the same position value but output X+P still have the embedding information. Why do different items in the embedding vector of a word need to have different positional values? $\endgroup$
    – kyc12
    Commented Nov 24, 2020 at 2:55
  • $\begingroup$ Because otherwise, you are deciding to waste the other $𝑑−1$ positions of the vector $\endgroup$
    – noe
    Commented Nov 24, 2020 at 8:10
  • $\begingroup$ ok, so the idea of 'trainable positional embeddings' helps me understand your answer. $\endgroup$
    – kyc12
    Commented Dec 3, 2020 at 23:19

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.