1
$\begingroup$

I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain it intuitively. Thanks in advance!

$\endgroup$

1 Answer 1

0
$\begingroup$

With relative position bias, you are extending the concept of self-attention to also encode the distance between any two tokens. Basically you let the model itself learn the relative distance between any 2 tokens instead of feeding that information yourself. Most of the time (as shown in the paper), the model does a good job at figuring out the relationships between various tokens. If that is the case, it is always better to make the model generic by giving it the flexibility rather than providing this data & forcing the model to use it. Usually generalized models perform better. This is the simplest reason to not feed explicit positional embeddings. For more details on the bias parameterization, you can refer to the relevant section in - https://towardsdatascience.com/swin-vision-transformers-hacking-the-human-eye-4223ba9764c3

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.