Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
51
votes
4answers
35k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
10
votes
2answers
4k views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
15
votes
1answer
6k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
12
votes
3answers
794 views

How does attention mechanism learn?

I know how to build an attention in neural networks. But I don’t understand how attention layers learn the weights that pay attention to some specific embedding. I have this question because I’m ...
0
votes
1answer
39 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...