66 votes

What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$...
Esmailian's user avatar
  • 9,147
54 votes
Accepted

What is the positional encoding in the transformer model?

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking ...
Batool's user avatar
  • 656
29 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Here's the list of difference that I know about attention (AT) and self-attention (SA). In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have ...
artoby's user avatar
  • 391
27 votes
Accepted

Why is the decoder not a part of BERT architecture?

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
noe's user avatar
  • 22.3k
22 votes
Accepted

Can BERT do the next-word-predict task?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you ...
noe's user avatar
  • 22.3k
22 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Let me try to keep it more intuitive and less mathematical Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give ...
Allohvk's user avatar
  • 848
16 votes

Gumbel-Softmax trick vs Softmax with temperature

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution $$...
Asi sheffer's user avatar
16 votes
Accepted

Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings ...
noe's user avatar
  • 22.3k
14 votes

What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
Juan Esteban de la Calle's user avatar
10 votes
Accepted

Variable input/output length for Transformer

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target ...
noe's user avatar
  • 22.3k
10 votes

Class token in ViT and BERT

My question is — why does this token exist as input in all the transformer blocks and is treated the same as the word / patches tokens? The transformers, by default are sequence to sequence networks. ...
Mohsen's user avatar
  • 101
9 votes
Accepted

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
noe's user avatar
  • 22.3k
9 votes

Why does the transformer positional encoding use both sine and cosine?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, ...
TitoOrt's user avatar
  • 1,842
8 votes

What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding. Here is the PE lookup table generation ...
Eris's user avatar
  • 81
8 votes
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
Jindřich's user avatar
  • 1,661
8 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
Darren Cook's user avatar
7 votes

How does attention mechanism learn?

From the Amazing Blog - FloydHub Blog- Attention Mechanisms Attention Mechanisms Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the ...
Pluviophile's user avatar
  • 3,550
6 votes
Accepted

How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
Esmailian's user avatar
  • 9,147
5 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
Jindřich's user avatar
  • 1,661
5 votes
Accepted

Can BERT be used for predicting words?

pip install transformers Then try this ...
ashutosh singh's user avatar
5 votes

Does BERT use GLoVE?

BERT's embeddings are 3 things : Token embeddings Segment embeddings Position embeddings I guess your question is about token embeddings. Token embeddings is a vector, where each token is encoded as ...
Astariul's user avatar
  • 984
5 votes
Accepted

Attention for time-series in neural networks

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the ...
Allohvk's user avatar
  • 848
4 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot This confused me very much at first because I was thinking of the ...
Denziloe's user avatar
  • 149
4 votes

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: ...
Simon Ren's user avatar
4 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are ...
Leevo's user avatar
  • 6,035
4 votes

What are good toy problems for testing Transformer architectures?

For MT, I always use the Multi30k dataset, English to German for debugging. It has only 30k sentences which are simple and template-like, with a correctly configured Transformer model, you should get ...
Jindřich's user avatar
  • 1,661
4 votes
Accepted

Does multi-head attention remove the need for self-attention?

Self-attention means X pays attention to X, as opposed to "normal" attention where ...
Darren Cook's user avatar
3 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary ...
mlstudent's user avatar
  • 131
3 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the word embedding; however, it is not actually. Adding new dimensions to ...
Hamid Mohammadi's user avatar
3 votes

Why is the decoder not a part of BERT architecture?

BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for ...
vishwajit kumar vishnu's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible