Why can't positions in transformers be simply appended to the input to preserve the positional information instead of using positional encodings?

Question

I saw in an intro to transformers in this video that positional encodings need to be used to preserve positional information, otherwise word order may not be understood by the neural network. They also explained why we cannot simply add positions to the original inputs or add other functions of positions to the input. However, what if we simply appended positions to the original input to preserve positional information? In fact, if directly using positional numbers does not suffice, couldn't they also just append positional encodings (containing the sine and cosine functions of differing frequencies) to the input? This would also solve the problem of loss of information which happens when one adds the positional encodings to the original input.

Answer 1

You could do that. I believe the authors of the original attention paper believed that it isn't as efficient as trying to inject the positional information into the embedding - you add parameters to the model and that has implications for training and inference speed.

I think the intuition is that the difference encoding between "foo" and "bar" after positional encoding becomes ("foo + pos_foo) - ("bar" + pos_bar) = ("foo" - "bar") + (pos_foo - pos_bar). So it should be possible for the model to learn pretty reasonably about how the difference in position modifies relationship between words.

It isn't the only or best way of adding positional info, as you allude to. In the conventional positional encoding, position information is fairly "absolute" which makes it hard to generalize to longer contexts and harms ability to learn that a) short distances probably imply more relationship than long and b) the relationship between words is probably similar across the context. Rotary positional encoding tries to address that.

Then there is ALiBi, which throws out the idea of embedding positional encoding entirely and just decays the strength of relationship based on distance, and that actually works OK. So, maybe positional info isn't that important to represent so exactly, let alone as additional dimensions in the embedding.

Answer 2

I understand that what you mean is the following:

1. Not using positional encodings.
2. Append some special tokens to the input sequence to represent the positions of the normal tokens within the sequence. For instance, the sequence ["my", "dog"] would be transformed into ["my", "dog", <0>, <1>]. This would imply that the position of "my" is 0 and the position of p "dog"p is 1. This association would be given by the position of the special tokens <0> and <1> within the sequence.

However, in a Transformer without the positional encodings, the model does not have any information about the token positions (which is precisely what gives meaning to our special tokens) and, therefore, it would not be able to establish the association between the positions of the special tokens <0> and <1> and their actual positions. This way, for the Transformer without positional encodings, the input ["my", "dog", <0>, <1>] and the input ["1", "my", "dog", <0>] would be exactly the same.