66
votes
What is the positional encoding in the transformer model?
For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$...
- 9,057
49
votes
Accepted
What is the positional encoding in the transformer model?
Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations:
Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings
Taking ...
- 606
27
votes
What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?
Here's the list of difference that I know about attention (AT) and self-attention (SA).
In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have ...
- 371
25
votes
Accepted
Why is the decoder not a part of BERT architecture?
The need for an encoder depends on what your predictions are conditioned on, e.g.:
In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
- 20.5k
22
votes
Accepted
Can BERT do the next-word-predict task?
BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.
BERT is trained on a masked language modeling task and therefore you ...
- 20.5k
21
votes
What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?
Let me try to keep it more intuitive and less mathematical
Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give ...
- 818
14
votes
Gumbel-Softmax trick vs Softmax with temperature
Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution
$$...
- 141
14
votes
What is the positional encoding in the transformer model?
Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle).
But you have to take ...
- 2,262
14
votes
Accepted
Transformer model: Why are word embeddings scaled before adding positional encodings?
This is specified in the original Transformer paper, at the end of section 3.4:
Transcription:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings ...
- 20.5k
9
votes
Accepted
Variable input/output length for Transformer
Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target ...
- 20.5k
9
votes
Class token in ViT and BERT
My question is — why does this token exist as input in all the
transformer blocks and is treated the same as the word / patches
tokens?
The transformers, by default are sequence to sequence networks. ...
- 91
8
votes
Why does the transformer positional encoding use both sine and cosine?
If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.
In order not to use a sequence of integers (1, ...
- 1,822
8
votes
Accepted
Does BERT use GLoVE?
BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
- 1,631
7
votes
What is the positional encoding in the transformer model?
To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding.
Here is the PE lookup table generation ...
- 71
7
votes
Accepted
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
- 20.5k
7
votes
How does attention mechanism learn?
From the Amazing Blog - FloydHub Blog- Attention Mechanisms
Attention Mechanisms
Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the ...
- 3,360
7
votes
Transformer model: Why are word embeddings scaled before adding positional encodings?
Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
- 807
6
votes
Accepted
How do attention mechanisms in RNNs learn weights for a variable length input
Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size.
For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
- 9,057
5
votes
Accepted
5
votes
Does BERT use GLoVE?
BERT's embeddings are 3 things :
Token embeddings
Segment embeddings
Position embeddings
I guess your question is about token embeddings.
Token embeddings is a vector, where each token is encoded as ...
- 984
5
votes
Accepted
Attention for time-series in neural networks
It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the ...
- 818
4
votes
ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)
The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here.
my original code is:
...
- 41
4
votes
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
- 1,631
4
votes
What are good toy problems for testing Transformer architectures?
For MT, I always use the Multi30k dataset, English to German for debugging. It has only 30k sentences which are simple and template-like, with a correctly configured Transformer model, you should get ...
- 1,631
3
votes
Gumbel-Softmax trick vs Softmax with temperature
From a practical and theoretical perspective, when is it beneficial to
incorporate Gumbel noise into a neural network, as opposed to just
using Softmax with temperature?
You don't necessarily ...
- 131
3
votes
What is the advantage of positional encoding over one hot encoding in a transformer model?
You are mixing two different concepts in the same question:
One hot encoding: approach to encode $n$ discrete tokens by having an $n$-dimensional vectors with all 0's except one 1. This can be used ...
- 20.5k
3
votes
What is the advantage of positional encoding over one hot encoding in a transformer model?
The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would ...
- 1,631
3
votes
What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?
In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary ...
- 131
3
votes
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
the first few bits of the embedding are completely unusable by the
network because the position encoding will distort them a lot
This confused me very much at first because I was thinking of the ...
- 139
3
votes
Accepted
Transformer decoder output - how is it linear?
I'm not quite sure how's the decoder output is flattened into a single vector
That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the ...
- 5,346
Only top scored, non community-wiki answers of a minimum length are eligible