I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices.

In the paper (Section 3.5), they mentioned

Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks.

My question is that if there is no recurrence, why not use One Hot Encoding. What is the advantage of using a sinusoidal positional encoding?


3 Answers 3


You are mixing two different concepts in the same question:

  • One hot encoding: approach to encode $n$ discrete tokens by having an $n$-dimensional vectors with all 0's except one 1. This can be used to encode the tokens them selves in networks with discrete inputs, but only if $n$ is not very large, as the amount of memory needed is very large. Transformers (and most other NLP neural models) use embeddings, not one-hot encoding. With embeddings, you have a table with $n$ entries, each of them being a vector of dimensionality $e$. In order to represent token $k$, you select the $k$th entry in the embedding table. The embeddings are trained with the rest of the network in the task.
  • Positional encoding: in recurrent networks like LSTMs and GRUs, the network processes the input sequentially, token after token. The hidden state at position $t+1$ depends on the hidden state from position $t$. This way, the network has a means to identify the relative positions of each token by accumulating information. However, in the Transformer, there is no built-in notion of the sequence of tokens. Positional encodings are the way to solve this issue: you keep a separate embedding table with vectors. Instead of using the token to index the table, you use the position of the token. This way, the positional embedding table is much smaller than the token embedding table, normally containing a few hundred entries. For each token in the sequence, the input to the first attention layer is computed by adding up the token embedding entry and the positional embedding entry. Positional embeddings can either be trained with the rest of the network (just like token embeddings) or pre-computed by the sinusoidal formula from (Vaswani et al., 2017); having pre-computed positional embeddings leads to less trainable parameters with no loss in the resulting quality.

Therefore, there is no advantage of anyone over the other, as they are used for orthogonal purposes.

  • $\begingroup$ This should be the accepted answer. If we consider N observations then there would be N entries at each of the D (vector length/dimension) positional locations. Then how is the positional encoding computed: is it the sum of all N? $\endgroup$ Feb 16 at 5:47
  • $\begingroup$ @WestCoastProjects that is not the standard case for positional encodings, so there is no standard way to apply positional encodings there. A customized approach must be tailored to it. $\endgroup$
    – noe
    Feb 16 at 7:32
  • $\begingroup$ I'm not understanding the encodings apparently. I'll look at the original paper again. $\endgroup$ Feb 16 at 7:48
  • $\begingroup$ Wait, I just re-read your previous answer and maybe I misunderstood it. Maybe you meant N independent observations in a single batch? Anyway, you can create a new question (instead of a comment) and provide all the details of your doubt. $\endgroup$
    – noe
    Feb 16 at 8:14

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions.

On the other hand paper on Convolutional Sequence to Sequence Learning published shortly before the Transformer uses one-hot encoding and learned embeddings for positions and it seems it does not make any harm there.

  • $\begingroup$ iiuc the positional embeddings are simply an additional piece of metadata associated with each token. They are not one-hot encoded . Also there is no difference in the algorithm structure for earlier vs later positions. If you are suggesting that the attention embeddings (not position embeddings since those are not learnt but simply assigned) are more accurate for earlier positions then please explain how that comes about. $\endgroup$ Feb 16 at 5:42

By just using one-hot encoding you get an output in range 0..1, which is not normal distributed, as it is expected by the first linear layer with uniform(-1/sqrt(in_features), 1/sqrt(in_features)) initialization, so you might get longer training time and worse generalization due to input distribution and weight initialization.

On the other hand, why you should use one-hot, when you can actually just add a linear layer without bias term? This is an embedding layer, that is normal-initialized with mean 0 and std 1. By getting something at some index you get uniform-shaped activations, unlike one-hot. Embeddings layer also can be used to reduce redundant dimensionality explosion caused by one-hot encoding. You may want consider freezing weights of embeddings layer, as you don't gain much from two linear layers (input x -> embeddings -> linear) without non-linearity between embeddings layer and linear layer.

And what I can say about positional encoding in a feedforward neural network, which is BERT, GPT and BARD. Take a look how positional encoding is encoding your inputs:

x = x + pe

By this logic you can pre-generate pe (positional encoding) tensor out of random beta U-shaped distribution. You cannot put many features into one feature by just summing your token embedding with position embedding and with segment embedding. That's why even with segment embeddings BERT needed separation token.

Concluding, positional information in feedforward neural network is carried by the neuron itself. If you just throw out positional encoding from BERT architecture you would not lose in performance.

To prove it, there is two important papers I have just discovered:

But where positional emdebbings might work and improve performance? Of course in convolutional networks, where you have kernels, not neurons. Positional embeddings might get injected as a channel to the input.

  • $\begingroup$ It seems you have a deep CNN architectures analysis background. This answer is not clear on why the positional encoding is the crucial factor that allows the Transformer architecture to dispense/remove the recurrent RNN or memory-based LSTM in favor of positionless fully-connected layer. The positional encoding is a crucial component to the Transformer since it is the only way in which the relative position of tokens is represented. I'm interested to see an update to your answer that describes that mechanism. $\endgroup$ Feb 16 at 5:51
  • $\begingroup$ I am happy to answer you, that there is two ablation study of positional encoding. Both studies concluded, that their model performance is not decreased, nor improved. I am referring to "Attention Is All You Need" Table 3 row (E) and "Convolutional Sequence to Sequence Learning" Table 4. Also, you say that fully-connected layer can be "positionless". Can you elaborate on that? If I take house price prediction FNN and swap two input features, i.e. # of bedrooms and age of estate, I will get the same prediction? $\endgroup$ Feb 16 at 6:19
  • $\begingroup$ @WestCoastProjects, I have just discovered following interesting papers, take a look: arxiv.org/abs/2203.16634 arxiv.org/abs/2305.19466 $\endgroup$ Feb 16 at 14:34
  • $\begingroup$ I'm working through the original paper again now and will look at your new links as well. $\endgroup$ Feb 16 at 16:38

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.