68 votes

What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$...
Esmailian's user avatar
  • 9,312
60 votes
Accepted

What is the positional encoding in the transformer model?

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking ...
Batool's user avatar
  • 716
22 votes
Accepted

Can BERT do the next-word-predict task?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you ...
noe's user avatar
  • 26.5k
17 votes

Is the Transformer decoder an autoregressive model?

The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time. The non-autoregressive training can be done because of two factors: We don't use the ...
noe's user avatar
  • 26.5k
16 votes

What is the use of [SEP] in paper BERT?

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence ...
Astariul's user avatar
  • 1,004
16 votes
Accepted

Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings ...
noe's user avatar
  • 26.5k
15 votes
Accepted

Is BERT a language model?

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes ...
noe's user avatar
  • 26.5k
14 votes

What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
Juan Esteban de la Calle's user avatar
14 votes
Accepted

What are the inputs to the first decoder layer in a Transformer model during the training phase?

Following your example: The source sequence would be How are you ...
noe's user avatar
  • 26.5k
12 votes
Accepted

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
noe's user avatar
  • 26.5k
11 votes
Accepted

what is the first input to the decoder in a transformer model?

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value (...
noe's user avatar
  • 26.5k
11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 26.5k
10 votes

Why does the transformer positional encoding use both sine and cosine?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, ...
TitoOrt's user avatar
  • 1,862
10 votes

Overfitting with text classification using Transformers

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...
Jindřich's user avatar
  • 1,741
9 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot This confused me very much at first because I was thinking of the ...
Denziloe's user avatar
  • 199
9 votes
Accepted

Why does vanilla transformer has fixed-length input?

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the ...
noe's user avatar
  • 26.5k
9 votes
Accepted

What is the difference between GPT blocks and Transformer Decoder blocks?

GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model: For GPT-2, this is ...
noe's user avatar
  • 26.5k
9 votes
Accepted

What is the difference between BERT and Roberta

The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1: BERT relies on randomly ...
noe's user avatar
  • 26.5k
9 votes
Accepted

Does fine-tuning require retraining the entire model?

No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
Andy's user avatar
  • 650
8 votes

What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding. Here is the PE lookup table generation ...
Eris's user avatar
  • 81
8 votes
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
Jindřich's user avatar
  • 1,741
8 votes

What's the right input for gpt-2 in NLP

Updated answer After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
noe's user avatar
  • 26.5k
8 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
Darren Cook's user avatar
8 votes

Is time series forecasting possible with a transformer?

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time ...
Brian O'Donnell's user avatar
8 votes
Accepted

Dimensions of Transformer - dmodel and depth

d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case ...
noe's user avatar
  • 26.5k
8 votes
Accepted

Do transformers (e.g. BERT) have an unlimited input size?

You are right that a transformer can take in an arbitrary amount of tokens even with fixed parameters, excluding the positional embedding matrix, whose size directly grows with the maximum allowed ...
noe's user avatar
  • 26.5k
7 votes
Accepted

Proper masking in the transformer model

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the ...
noe's user avatar
  • 26.5k
7 votes

ChatGPT: How to use long texts in prompt?

I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it. Boring answer #1 ...
Valentas's user avatar
  • 1,209
7 votes

Further Training a pre-trained LLM

Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach. The LLM I used was GPT-J 6B to generate MCQ's. Some ...
spectre's user avatar
  • 2,065
6 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

The confusion here is that we believe positional embedding is a more complicated version of adding positional information to the word embedding; however, it is not actually. Adding new dimensions to ...
Hamid Mohammadi's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible