Skip to main content
3 votes
Accepted

Cross-attention mask in Transformers

From the paper: "We also modify the self-attention sub-layer in the decoder stack to prevent positions from attending to subsequent positions. This masking, combined with fact that the output ...
ИванКарамазов's user avatar
3 votes

Cross-attention mask in Transformers

I don't understand if we should combine the causal mask with the padding mask from the encoder output or if we should just apply the padding mask (since the VALUES are coming from the encoder, and we ...
Valentin Calomme's user avatar
2 votes

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409 For me, the takeaways were: The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would ...
Darren Cook's user avatar
  • 1,074
2 votes
Accepted

Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?

I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly ...
Alexander Wan's user avatar
2 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I'm a PhD student in natural language processing, and I hope I can clear up some of the terminology used in previous answers to this question in a way that's helpful for full understanding. To clarify ...
Pro Q's user avatar
  • 195
2 votes

How can we use a transfomer model with new data if we still don't have the output?

Note: this answer assumes that the question is about a scenario were there is no output data available. The Transformer model is typically trained using supervised ...
noe's user avatar
  • 27k
2 votes
Accepted

Confused about Q and K in Attention Mechanism

First of all, it is necessary to figure out what kind of attention mechanism we are talking about. If this is a «classical» attention mechanism, then Q and K are calculated for completely different ...
Gromov's user avatar
  • 126
2 votes
Accepted

Practical Experiments on Self-Attention Mechanisms: QQ^T vs. QK^T

QQᵀ only carries half the information of QKᵀ: ...
Darren Cook's user avatar
  • 1,074
2 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

The best answer I have seen is this Reddit answer by pappypapaya: In attention, we basically take two word embeddings (x and y), pass one through a Query transformation matrix (Q) and the second ...
Bill Vander Lugt's user avatar
2 votes
Accepted

How can we use a transfomer model with new data if we still don't have the output?

Note: this answer assumes that the question is about how to use the Transformer model at inference if there is no output to use At training time, we have the ...
noe's user avatar
  • 27k
1 vote

Why do we use similarity/cosine between Query and Key in attention?

the purpose of this similarity is not to find the most similar words in terms of their meaning, but rather to identify which words in the input sequence are most relevant for generating each output ...
Amritesh Nandan's user avatar
1 vote

Why do we use similarity/cosine between Query and Key in attention?

Remember that the embedding representations change as the input goes through the model. For the very first input (after embedding tokens), you have an embedding sequence where each item represents a ...
Karl's user avatar
  • 756
1 vote

How can we use a transfomer model with new data if we still don't have the output?

This answer is based on Eduardo Munoz's blog "Attention is all you need: Discovering the Transformer paper" in Towards Data Science. To understand how transformer prediction works, the main ...
Lynn's user avatar
  • 1,307
1 vote

Understanding Multi-headed Attention from architecture details

No. As shown in the original Transformer paper that you linked, the results of the individual heads are concatenated into a single vector but then they pass through another linear layer, which does ...
noe's user avatar
  • 27k
1 vote

Tensorflow diagram for attention mechanism

This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure). The ingoing arrows are inputs to the ...
noe's user avatar
  • 27k
1 vote

What is the advantage of positional encoding over one hot encoding in a transformer model?

By just using one-hot encoding you get an output in range 0..1, which is not normal distributed, as it is expected by the first linear layer with ...
Ivan Stepanov's user avatar
1 vote

What does it mean order of input sequence does not matter for transformer self-attention head?

Your code sample works because you are using two different tokens to do the prediction of "!" and "?". The model is trained to transform the token "A" to "?" ...
yuitora's user avatar
  • 11
1 vote

What does it mean order of input sequence does not matter for transformer self-attention head?

+1 for a great reproducible code sample. The answer to your main question can be seen by looking at the linear algebra, of what self-attention does. Go back to what matrix multiplication is doing, and ...
Darren Cook's user avatar
  • 1,074
1 vote

What does it mean order of input sequence does not matter for transformer self-attention head?

When the order of input sequence is changed, the positions of self attention values change (rows interchanged), but the values remain same (like you mentioned head.forward('AB')='BC' and head.forward('...
shivani's user avatar
  • 140
1 vote

query, key and value interpretation in transformers ( encoder - decoder framework )

Note three things: The output of the encoder is not English but just what the decoder needs from the source sentence to generate the translation. Only the first decoder layer receives the target ...
noe's user avatar
  • 27k
1 vote
Accepted

Why are they called Key,Value,Query-Vectors when they are actually matrices

Those matrices can be seen as N vectors, each one of length embed_size, where N = sequence_length.
noe's user avatar
  • 27k
1 vote

How many parameters does the vanilla Transformer have?

Thanks to Bruno Lubascher's answer I asked ChatGPT and double-checked its answer. The fixed formula was: $V \times d_{model} + V \times d_{model} + N \times (2 \times h \times 3 \times d_{model} \...
Jill-Jênn Vie's user avatar
1 vote

How do attention mechanism in CNN for images?

Attention mechanisms in the context of image processing using Convolutional Neural Networks (CNNs) are used to allow the model to focus on specific parts of the image that are more relevant for the ...
Harshad Patil's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible