0
$\begingroup$

The transformer architecture has no sense of the relative positions of the word and hence we need to pass that information apriori to the along with the word embeddings to the model

The positional encoding currently implemented in the transformers uses trig enccoding

Why is a simple positional encoding scheme like concatenating 1/n, 2/n, ... 2048/n (for gpt-4) to the input words not chosen? What are the arguments against using it?

$\endgroup$
2
  • $\begingroup$ Usually, when dealing with neural network design, the arguments for using some approach are whether an approach works or not, and whether it works better than the other proposed alternatives. Then, post-hoc explanations. $\endgroup$
    – noe
    Commented Oct 9, 2023 at 7:45
  • $\begingroup$ hmm.. I figured as much. I was thinking in case there is some explanation that I am not aware of except for "It has been found to work better than the alternatives" $\endgroup$ Commented Oct 9, 2023 at 7:52

1 Answer 1

0
$\begingroup$

Please see the link which compares the design choice of positional encoding and the requirements. It will answer your question. https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3

$\endgroup$
2
  • $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Commented Jan 3 at 11:34
  • $\begingroup$ Your answer could be improved with additional supporting information. Please edit to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers in the help center. $\endgroup$
    – Community Bot
    Commented Jan 5 at 18:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.