3
votes
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the word embedding; however, it is not actually. Adding new dimensions to ...
2
votes
Transformers doubt
We should not mistake the K, Q and V vectors received by the multi-head attention block with those received by the scaled dot-product block.
The K, Q and V vectors that are fed to the multi-head ...
- 23.8k
2
votes
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
Take a look at the ALiBi paper: https://arxiv.org/abs/2108.12409
For me, the takeaways were:
The sin/cos idea in the "Attention is All You Need" added complexity in the hope it would ...
- 967
2
votes
Accepted
Why cant we use normalise position encodings instead of the cos and sine encodings used in the Transformer paper?
I found this post really helpful for understanding some of the nice properties behind positional embeddings. I'll give a short summary of the relevant portions of the post in my answer, but I highly ...
- 156
2
votes
How can we use a transfomer model with new data if we still don't have the output?
Note: this answer assumes that the question is about a scenario were there is no output data available.
The Transformer model is typically trained using supervised ...
- 23.8k
2
votes
Accepted
How can we use a transfomer model with new data if we still don't have the output?
Note: this answer assumes that the question is about how to use the Transformer model at inference if there is no output to use
At training time, we have the ...
- 23.8k
1
vote
Understanding Multi-headed Attention from architecture details
No.
As shown in the original Transformer paper that you linked, the results of the individual heads are concatenated into a single vector but then they pass through another linear layer, which does ...
- 23.8k
1
vote
How can we use a transfomer model with new data if we still don't have the output?
This answer is based on Eduardo Munoz's blog "Attention is all you need: Discovering the Transformer paper" in Towards Data Science.
To understand how transformer prediction works, the main ...
- 1,176
1
vote
Tensorflow diagram for attention mechanism
This is a diagram of the attention layer that appears in the English version of the Tensorflow Transformer tutorial (other languages do not have this figure).
The ingoing arrows are inputs to the ...
- 23.8k
1
vote
Accepted
Why do position embeddings work?
The token embeddings are not fixed, they are learned. Therefore, during training, the value learned for the token embeddings is intrinsically one that is useful after adding it up with the positional ...
- 23.8k
1
vote
query, key and value interpretation in transformers ( encoder - decoder framework )
Note three things:
The output of the encoder is not English but just what the decoder needs from the source sentence to generate the translation.
Only the first decoder layer receives the target ...
- 23.8k
1
vote
Accepted
Why are they called Key,Value,Query-Vectors when they are actually matrices
Those matrices can be seen as N vectors, each one of length embed_size, where N = sequence_length.
- 23.8k
1
vote
Accepted
computer vision transformers: ViT does not have a decoder?
It depends on the task you want to perform. The goal here is to find a way to represent your image as a sequence of embedding vectors representing each patch of the image
Once you have obtained an ...
- 151
1
vote
Below text-classification model gives accuracy of 0.77 only on one dataset and 0.99 on spam-ham dataset? What should I do to increase with my dataset?
Your question can be given a general answer along the lines of the comment from @Erwan . Scoring highly on a chosen metric (and as he mentioned accuracy is only one type and possibly not the best one) ...
1
vote
How many parameters does the vanilla Transformer have?
Thanks to Bruno Lubascher's answer I asked ChatGPT and double-checked its answer.
The fixed formula was:
$V \times d_{model} + V \times d_{model} + N \times (2 \times h \times 3 \times d_{model} \...
- 111
1
vote
Accepted
How many parameters does the vanilla Transformer have?
Table 3 has all the values of the hyper-parameters of the models.
See the image below, green are for the base and blue for the big model.
You can use these to get the matrices sizes. For example for ...
- 3,518
1
vote
Accepted
Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?
The training of a self-attention layer will result in the update of the $W$ matrices and the gradient being propagated back to the previous layer.
At the end of the self-attention blocks, the back-...
- 23.8k
1
vote
Vision Transformer ViT Parameter count
My calculation was based on a wrong understanding of the self attention mechanism. In Attention is all you need the authors point out that they won't use the full $768 \times 768$ matrices when they ...
- 111
1
vote
Why is the decoder not a part of BERT architecture?
In short, Bidirectional Encoder Representations from Transformers (BERT) is not designed for decorder-related tasks.
I can't see how BERT makes predictions without using a decoder unit, which was a ...
- 11
1
vote
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
The following is conjecture, not fact.
If you look at how much each scalar in the the positional embedding vector changes as a function of position... you'll find that many of the scalars barely ...
- 131
Only top scored, non community-wiki answers of a minimum length are eligible