At least in the first self-attention layer in the encoder, inputs have a correspondence with outputs, I have the following questions.

  • Isn't ordering already implicitly captured by the query vectors, which themselves are just transformations of the inputs?
  • What do the sinusoidal positional encodings capture that the ordering of the query vectors don't already do?
  • Am I perhaps mistaken in thinking that transformers take in the entire input at once?
  • How are words fed in?
  • If we feed in the entire sentence at once, shouldn't the ordering be preserved?

source: https://web.eecs.umich.edu/~justincj/slides/eecs498/FA2020/598_FA2020_lecture13.pdf

  • 1
    $\begingroup$ I stumbled upon the same question. To me it also looks like the attention mechanism has all "information about the order" it needs. Maybe the positional encoding is for the Feed Forward part, that processes all tokens in parallel and thus does not have any information about position it may need? $\endgroup$
    – Stiefel
    Jan 2, 2023 at 23:01
  • $\begingroup$ I think you are confused with "if a model takes into account the sequence order" with "if we feed in the sequence at once". $\endgroup$
    – chichi
    May 28, 2023 at 21:47

1 Answer 1


Consider the input sentence - "I am good".

In RNNs, we feed the sentence to the network word by word. That is, first the word "I" is passed as input, next the word "am" is passed, and so on. We feed the sentence word by word so that our network understands the sentence completely.

But with the transformer network, we don't follow the recurrence mechanism. So, instead of feeding the sentence word by word, we feed all the words in the sentence parallel to the network. Feeding the words in parallel helps in decreasing the training time and also helps in learning the long-term dependency.

We feed the words parallel to the transformer, the word order (position of the words in the sentence) is important. So, we should give some information about the word order to the transformer so that it can understand the sentence.

If we pass the input matrix directly to the transformer, it cannot understand the word order. So, instead of feeding the input matrix directly to the transformer, we need to add some information indicating the word order (position of the word) so that our network can understand the meaning of the sentence. To do this, we introduce a technique called positional encoding. Positional encoding, as the name suggests, is an encoding indicating the position of the word in a sentence (word order).

  • $\begingroup$ Correct me if I'm wrong, but isn't the only parallel part about transformers the multi-headed attention? As in, within each head, words can't be fed in parallel, because they need to be there for context. $\endgroup$
    – Cole
    Oct 18, 2021 at 18:42
  • 4
    $\begingroup$ The entire network is parallel, which is the big advantage of it. It is all just a series of matrix multiplications, with no recurrent processing. This drastically increased the training speed over RNNs, but the self attention mechanism also performs better in many domains it seems. So, without position, the network 'sees' the input as a bag of words. Just because you order them when setting up the input, doesn't mean the network 'cares' or 'understands' that your ordering matters. Sinusoidal encoding basically imparts a cosine distance over the input, making it ordinal. $\endgroup$ Oct 19, 2021 at 0:21
  • 2
    $\begingroup$ More info on how and why of positional encoding can be found kazemnejad.com/blog/… $\endgroup$ Apr 18, 2022 at 5:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.