Questions tagged [attention-mechanism]
The attention-mechanism tag has no usage guidance.
5
questions
51
votes
4answers
35k views
What is the positional encoding in the transformer model?
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:
I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
10
votes
2answers
4k views
Variable input/output length for Transformer
I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site
http://jalammar.github.io/illustrated-transformer/ which provided a great ...
15
votes
1answer
6k views
Can BERT do the next-word-predict task?
As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
12
votes
3answers
794 views
How does attention mechanism learn?
I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding.
I have this question because I’m ...
0
votes
1answer
39 views
How do the linear layers in the attention mechanism work?
I think I now the answer to my question but I dont really get confirmation.
When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
