what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that gives context based on postion of word in sentence "* defintion for attention vector was give as "For ever word we can have attention vector generated which captures contextual relationship between words in sentence"

Capturing context information based on distance(postional vector) and attention (attention vector ) sounds same right? or is it different ?


1 Answer 1


So the question asks between the difference between an attention vector and a positional vector.

To answer this question, will give some context into how the transformer differs from a sequential model, such as RNNs and LSTMs. In the case of RNNs and LSTMs, data is fed sequentially "one-by-one" into the model to predict the output (whether that is at each time step or after observing the whole sequence. This is irrelevant though in the context of the question.)

In a transformer model, the whole sequence is fed into the model, just like as you would with a conventional neural network. However, the problem is that, unlike with RNNs/LSTMs, there is no way for the transformer model to understand the ordering of the instances in the sequence as the whole sequence is fed into the model. Therefore, we need positional embeddings (positional vector, in your terminology) to add information to the individual instances which tells the model the ordering of the instances in the sequence.

Attention, in the context of transformers, works on the basis that it assigns higher coefficients to instances in the sequences which are most relevant to decoding the hidden representation from the encoder. Unlike a basic encoder-decoder model, with attention, we are able to flexibly assign which input instances in the sequences have the most "say" in predicting the next output instance in the output sequence.

I hope this clarifies some understanding. If not, there is a great article on transformers here: http://www.peterbloem.nl/blog/transformers

  • $\begingroup$ wow i got ti now thanks , i read more about transoformers and i got 1 more query btw do you how excatly is the first word for french sentence is found while translating sentence from english to french using transformers ? , i saw translation case was commonly used reference to explain transformers in many articles . $\endgroup$
    – star
    Jul 4, 2020 at 17:07
  • $\begingroup$ Any chance you could rephrase the question? Having trouble understanding what you are asking. $\endgroup$
    – shepan6
    Jul 4, 2020 at 17:32
  • $\begingroup$ Consider the language translation task using transformers, the encoder takes english sentence eg: "i am going to study" and gives it to decoder which will output the french sentence " je vais étudier" . i want to understand how exactly the encoder is able to find the first french word ("je" in this case ) using the english sentence where is this mapping happening in transformers ? $\endgroup$
    – star
    Jul 4, 2020 at 17:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.