I am trying to make a model that uses a Transformer to see the relationship between several data vectors but the order of the data is not relevant in this case, so I am not using the PositionalEncoding.

Since the performance of models using Transformers is quite improved with the use of this part do you think that if I remove that part I am breaking the potential of Transformers or is it correct to do so?

  • $\begingroup$ If the order doesn't matter, then you should be fine. If you aren't trying predict ordered out in particular, it will just look like a bag of words to the Transformer and you won't be using any look-ahead masking or anything so you should be fine. All the Transformer is doing at that point is creating weighted scores among your input tokens to reveal relationships, unordered in this case. $\endgroup$ Nov 14 at 19:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.