1
$\begingroup$

I saw in an intro to transformers in this video that positional encodings need to be used to preserve positional information, otherwise word order may not be understood by the neural network. They also explained why we cannot simply add positions to the original inputs or add other functions of positions to the input. However, what if we simply appended positions to the original input to preserve positional information? In fact, if directly using positional numbers does not suffice, couldn't they also just append positional encodings (containing the sine and cosine functions of differing frequencies) to the input? This would also solve the problem of loss of information which happens when one adds the positional encodings to the original input.

$\endgroup$
2
  • $\begingroup$ The video you linked is about normal transformers. There is no mention of CNNs in that video. $\endgroup$
    – noe
    Commented Apr 19, 2022 at 14:14
  • $\begingroup$ @noe thanks for notifying me about the issue, I was actually reading a paper on a CNN based on transformers and hence the mistake. I have edited the question accordingly. $\endgroup$ Commented Apr 21, 2022 at 15:24

2 Answers 2

1
$\begingroup$

You could do that. I believe the authors of the original attention paper believed that it isn't as efficient as trying to inject the positional information into the embedding - you add parameters to the model and that has implications for training and inference speed.

I think the intuition is that the difference encoding between "foo" and "bar" after positional encoding becomes ("foo + pos_foo) - ("bar" + pos_bar) = ("foo" - "bar") + (pos_foo - pos_bar). So it should be possible for the model to learn pretty reasonably about how the difference in position modifies relationship between words.

It isn't the only or best way of adding positional info, as you allude to. In the conventional positional encoding, position information is fairly "absolute" which makes it hard to generalize to longer contexts and harms ability to learn that a) short distances probably imply more relationship than long and b) the relationship between words is probably similar across the context. Rotary positional encoding tries to address that.

Then there is ALiBi, which throws out the idea of embedding positional encoding entirely and just decays the strength of relationship based on distance, and that actually works OK. So, maybe positional info isn't that important to represent so exactly, let alone as additional dimensions in the embedding.

$\endgroup$
0
$\begingroup$

I understand that what you mean is the following:

  1. Not using positional encodings.
  2. Append some special tokens to the input sequence to represent the positions of the normal tokens within the sequence. For instance, the sequence ["my", "dog"] would be transformed into ["my", "dog", <0>, <1>]. This would imply that the position of "my" is 0 and the position of p "dog"p is 1. This association would be given by the position of the special tokens <0> and <1> within the sequence.

However, in a Transformer without the positional encodings, the model does not have any information about the token positions (which is precisely what gives meaning to our special tokens) and, therefore, it would not be able to establish the association between the positions of the special tokens <0> and <1> and their actual positions. This way, for the Transformer without positional encodings, the input ["my", "dog", <0>, <1>] and the input ["1", "my", "dog", <0>] would be exactly the same.

$\endgroup$
2
  • $\begingroup$ My understanding is that in NLP, words such as "my" and "dog" would be passed down as vectors of numbers which indicate how close they are to other words in dictionary. If the positions of those words are appended to those vectors would the transformer still not be able to decipher the position? For example, if "my" is encoded as [0,1,0.5] and "dog" is encoded as [0.4, 0.7, 1], couldn't we just pass as input [0,1,0.5,0] and [0.4,0.7,1,1] to the transformer? $\endgroup$ Commented May 4, 2022 at 11:38
  • $\begingroup$ If you concatenate positional encoding to the end of embedding vectors, for example, then no those two inputs are not the same at all. That is not the reason. $\endgroup$
    – Sean Owen
    Commented Jul 16, 2023 at 13:24

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.