These past days I have started learning about transformers, but there's something that I don't really get. From what I read, static positional encoding helps us adding positional information in a sequence. And one of the ways to get it is using the following "formula":
What I want to know is that if this positional encoding depends on what is in the sequence, the content of the sequence? Or it is something independent from what is inside the sequence vector? Like, if I have two sequences, for example: [ 12., -378., 0.] and [ 96., 66., 0.], the positional encoding of both would be the same? Is that's the case, what do we do to differentiate between these two sequences? I read somewhere that we sum the embedding and the positional encoding, but I am not sure. Maybe it's a really stupid doubt, but I want to get this concept clear. Thanks!
