66
votes
What is the positional encoding in the transformer model?
For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be
$$...
- 9,147
54
votes
Accepted
What is the positional encoding in the transformer model?
Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations:
Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings
Taking ...
- 656
22
votes
Accepted
Can BERT do the next-word-predict task?
BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling.
BERT is trained on a masked language modeling task and therefore you ...
- 22.3k
16
votes
Accepted
Transformer model: Why are word embeddings scaled before adding positional encodings?
This is specified in the original Transformer paper, at the end of section 3.4:
Transcription:
3.4 Embeddings and Softmax
Similarly to other sequence transduction models, we use learned embeddings ...
- 22.3k
15
votes
Accepted
Is BERT a language model?
No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM.
A normal LM takes ...
- 22.3k
15
votes
Is the Transformer decoder an autoregressive model?
The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time.
The non-autoregressive training can be done because of two factors:
We don't use the ...
- 22.3k
14
votes
What is the positional encoding in the transformer model?
Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle).
But you have to take ...
- 2,262
14
votes
What is the use of [SEP] in paper BERT?
First let's understand why the format is like this.
BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence ...
- 984
12
votes
Accepted
What are the inputs to the first decoder layer in a Transformer model during the training phase?
Following your example:
The source sequence would be How are you ...
- 22.3k
11
votes
Accepted
what is the first input to the decoder in a transformer model?
At each decoding time step, the decoder receives 2 inputs:
the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value (...
- 22.3k
10
votes
Accepted
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
- 22.3k
9
votes
Accepted
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
- 22.3k
9
votes
Why does the transformer positional encoding use both sine and cosine?
If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input.
In order not to use a sequence of integers (1, ...
- 1,842
9
votes
Accepted
Why does vanilla transformer has fixed-length input?
The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it.
The amount of memory needed by the self-attention in the ...
- 22.3k
9
votes
Overfitting with text classification using Transformers
Your model is overfitting. You should try standard methods people use to prevent overfitting:
Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...
- 1,661
9
votes
Accepted
What is the difference between GPT blocks and Transformer Decoder blocks?
GPT uses an unmodified Transformer decoder, except that it lacks the encoder attention part. We can see this visually in the diagrams of the Transformer model and the GPT model:
For GPT-2, this is ...
- 22.3k
9
votes
Accepted
Does fine-tuning require retraining the entire model?
No, you don't need to retrain the entire model. Fine-tuning refers to taking the weights trained in the general model and then continuing training a bit using your specific data. Using this approach, ...
- 650
8
votes
What is the positional encoding in the transformer model?
To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think). They did not come up with the encoding.
Here is the PE lookup table generation ...
- 81
8
votes
Accepted
Does BERT use GLoVE?
BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
- 1,661
8
votes
What's the right input for gpt-2 in NLP
Updated answer
After reading @Jessica's answer, I carefully read the original GPT-2 paper and I confirm that the authors do not add special tokens, but simply the text ...
- 22.3k
8
votes
Transformer model: Why are word embeddings scaled before adding positional encodings?
Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
- 912
8
votes
Accepted
What is the difference between BERT and Roberta
The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1:
BERT relies on randomly ...
- 22.3k
7
votes
Accepted
Proper masking in the transformer model
I will take as reference fairseq's implementation of the Transformer model. With this assumption:
In the transformer, masks are used for two purposes:
Padding: in the multi-head attention, the ...
- 22.3k
7
votes
Is time series forecasting possible with a transformer?
Transformers can be used for time series forecasting. See the following articles:
Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al.
Deep Transformer Models for Time ...
- 276
7
votes
Accepted
Dimensions of Transformer - dmodel and depth
d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case ...
- 22.3k
7
votes
ChatGPT: How to use long texts in prompt?
I didn't find the site you mention that useful - perhaps it is not using the latest GPT model? Or ChatGPT does not yet have very good ability to understand the pdf file I provided it.
Boring answer #1 ...
- 1,074
6
votes
Accepted
Bert for QuestionAnswering input exceeds 512
The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
- 22.3k
6
votes
Variational Autoencoders VS Transformers
Variational AutoEncoder
VAE is an autoencoder whose encodings distribution is regularised during the training in order to ensure that its latent space has good properties allowing us to generate some ...
- 3,550
6
votes
Accepted
Do transformers (e.g. BERT) have an unlimited input size?
You are right that a transformer can take in an arbitrary amount of tokens even with fixed parameters, excluding the positional embedding matrix, whose size directly grows with the maximum allowed ...
- 22.3k
6
votes
Further Training a pre-trained LLM
Yes you are on the right track. What you are mentioning is called fine tuning the model. I personally have done this and used the same approach.
The LLM I used was GPT-J 6B to generate MCQ's. Some ...
- 1,876
Only top scored, non community-wiki answers of a minimum length are eligible