The need for positional encoding in transformer models is justified by permutation invariance of self-attention heads, because, without it, transformer wouldn't have any mechanism to take into account the order of the words.
Suppose we trained a simple trigram transformer model without positional encoding to predict C as the next token, if the input is AB. Because the self-attention head output has T (time) dimension, we actually predicted BC, in other words,
head.forward('AB')='BC'. We use only the last token C as the prediction of our model.
Now, because of the head permutation invariance,
head.forward('BA')='CB'. Thus, the next token prediction becomes B (last token in output of 'forward`). So why is it said that transformer models without positional encoding are position agnostic. In the above example, we permuted input tokens and obtained a different prediction (B instead of C)
UPDATE: made a colab notebook illustrating the concept: https://colab.research.google.com/drive/1ItIQTCg3sVGRUIGbrh1pxTNlfLU5C7kq?usp=sharing