I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that it is a function only of $\phi$, and not of $t$.
However, I'm still not clear on how exactly is this helping the model to attend to phrases such as "they are", no matter if they appear in position 3 or position 7 in the sentence. The attention layer gets as an input [$E(they)+PE(3)$, $E(are)+PE(4)$] where $PE(4)=T_1PE(3)$ and $T_1$ is a nice diagonal linear transformation that depends only on the position diff which is 1. How is it using those nice properties for learning and generalizing for the case where the same phrase "they are" appears in position 7? how is the fact that the linear transformation is nice and diagonal helping the model?