Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
35
votes
3answers
25k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
14
votes
1answer
5k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?