Skip to main content
Share Your Experience: Take the 2024 Developer Survey

Questions tagged [attention-mechanism]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
1 answer
3k views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
T Piper's user avatar
  • 123
104 votes
4 answers
111k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
Peyman's user avatar
  • 1,165
39 votes
8 answers
11k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
FremyCompany's user avatar
21 votes
1 answer
12k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
CoderOnly's user avatar
  • 711
13 votes
2 answers
9k views

Variable input/output length for Transformer

I was reading the paper "Attention is all you need" (https://arxiv.org/pdf/1706.03762.pdf ) and came across this site http://jalammar.github.io/illustrated-transformer/ which provided a great ...
Sean Lee's user avatar
  • 251
6 votes
1 answer
2k views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
Ian's user avatar
  • 63
2 votes
1 answer
680 views

Can the attention mask hold values between 0 and 1?

I am new to attention-based models and wanted to understand more about the attention mask in NLP models. attention_mask: an optional torch.LongTensor of shape [...
neel g's user avatar
  • 227
2 votes
1 answer
594 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
erre4's user avatar
  • 95
2 votes
3 answers
8k views

What is the advantage of positional encoding over one hot encoding in a transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, they used positional encoding with sin for even indices and cos for odd indices. In the paper (Section 3.5), they ...
Inderpartap Cheema's user avatar
1 vote
2 answers
303 views

Is a dense layer required for implementing Bahdanau attention?

I saw that everyone adds Dense( ) layer in their custom Bahdanau attention layer, which I think isn't needed. This is an image from a tutorial here. Here, we are just multiplying 2 vectors and then ...
Lucky Man's user avatar
1 vote
1 answer
848 views

Why does a decoder generate all hidden states during inference?

Seems that in Vanilla transformers at least (a la AIAYN), during inference time, the hidden states are generated for all tokens in the input sequence, but only the last one is used to predict the next ...
dashnick's user avatar
  • 163
1 vote
1 answer
851 views

why multiple attention heads learn differently

In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
Sandeep Bhutani's user avatar
1 vote
1 answer
260 views

Why and how BERT can learn different attentions for each head?

https://towardsdatascience.com/deconstructing-bert-distilling-6-patterns-from-100-million-parameters-b49113672f77 I read the blog above. It visualizes that different color/head has different ...
CoderOnly's user avatar
  • 711
1 vote
1 answer
4k views

What would be the target input for Transformer Decoder during test phase?

The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer I am having confusion about what ...
Hari Krishnan's user avatar