1
$\begingroup$

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that it is a function only of $\phi$, and not of $t$.

However, I'm still not clear on how exactly is this helping the model to attend to phrases such as "they are", no matter if they appear in position 3 or position 7 in the sentence. The attention layer gets as an input [$E(they)+PE(3)$, $E(are)+PE(4)$] where $PE(4)=T_1PE(3)$ and $T_1$ is a nice diagonal linear transformation that depends only on the position diff which is 1. How is it using those nice properties for learning and generalizing for the case where the same phrase "they are" appears in position 7? how is the fact that the linear transformation is nice and diagonal helping the model?

Improve this question
$\endgroup$
1
  • $\begingroup$ it can diminish irrelevant information like absolute position and consider only relative positioning, in this case "they are" functions as a unit no matter if on 1st or 7th position, whereas other models mix relevant relative info with irrelevant global info (which increases complexity without actually increasing performance). The art is to know what to include and what to leave out for a certain task and this model seems to leave out exactly what needed $\endgroup$
    – Nikos M.
    Commented Jan 17, 2021 at 8:25

0

Reset to default

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.