What is the advantage of positional encoding over one hot encoding in a transformer model?

Question

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices.

In the paper (Section 3.5), they mentioned

Since our model contains no recurrence and no convolution, in order for the model to make use of the order of the sequence, we must inject some information about the relative or absolute position of the tokens in the sequence. To this end, we add "positional encodings" to the input embeddings at the bottoms of the encoder and decoder stacks.

My question is that if there is no recurrence, why not use One Hot Encoding. What is the advantage of using a sinusoidal positional encoding?

Answer 1

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions.

On the other hand paper on Convolutional Sequence to Sequence Learning published shortly before the Transformer uses one-hot encoding and learned embeddings for positions and it seems it does not make any harm there.

Answer 2

You are mixing two different concepts in the same question:

• One hot encoding: approach to encode $n$ discrete tokens by having an $n$-dimensional vectors with all 0's except one 1. This can be used to encode the tokens them selves in networks with discrete inputs, but only if $n$ is not very large, as the amount of memory needed is very large. Transformers (and most other NLP neural models) use embeddings, not one-hot encoding. With embeddings, you have a table with $n$ entries, each of them being a vector of dimensionality $e$. In order to represent token $k$, you select the $k$th entry in the embedding table. The embeddings are trained with the rest of the network in the task.
• Positional encoding: in recurrent networks like LSTMs and GRUs, the network processes the input sequentially, token after token. The hidden state at position $t+1$ depends on the hidden state from position $t$. This way, the network has a means to identify the relative positions of each token by accumulating information. However, in the Transformer, there is no built-in notion of the sequence of tokens. Positional encodings are the way to solve this issue: you keep a separate embedding table with vectors. Instead of using the token to index the table, you use the position of the token. This way, the positional embedding table is much smaller than the token embedding table, normally containing a few hundred entries. For each token in the sequence, the input to the first attention layer is computed by adding up the token embedding entry and the positional embedding entry. Positional embeddings can either be trained with the rest of the network (just like token embeddings) or pre-computed by the sinusoidal formula from (Vaswani et al., 2017); having pre-computed positional embeddings leads to less trainable parameters with no loss in the resulting quality.

Therefore, there is no advantage of anyone over the other, as they are used for orthogonal purposes.