63 votes

What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$...
  • 8,667
34 votes
Accepted

What is the positional encoding in the transformer model?

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking ...
  • 456
21 votes
Accepted

Can BERT do the next-word-predict task?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you ...
  • 16.3k
15 votes
Accepted

Is BERT a language model?

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes ...
  • 16.3k
12 votes

What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
11 votes
Accepted

what is the first input to the decoder in a transformer model?

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value (...
  • 16.3k
11 votes

What is the use of [SEP] in paper BERT?

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence ...
  • 879
10 votes
Accepted

Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings ...
  • 16.3k
9 votes

Overfitting with text classification using Transformers

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...
  • 1,426
8 votes
Accepted

Why does vanilla transformer has fixed-length input?

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the ...
  • 16.3k
7 votes
Accepted

Proper masking in the transformer model

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the ...
  • 16.3k
7 votes

Why does the transformer positional encoding use both sine and cosine?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, ...
  • 1,772
6 votes

What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the ...
  • 61
6 votes
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
  • 1,426
6 votes
Accepted

Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
  • 16.3k
6 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
  • 16.3k
5 votes
Accepted

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
  • 16.3k
5 votes

BPE vs WordPiece Tokenization - when to use / which?

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end ...
  • 16.3k
5 votes
Accepted

What is the difference between BERT architecture and vanilla Transformer architecture

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder BERT ends with the final representation of the words after ...
  • 683
5 votes

Is time series forecasting possible with a transformer?

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time ...
5 votes
Accepted

What is the difference between BERT and Roberta

The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1: BERT relies on randomly ...
  • 16.3k
5 votes

Is the Transformer decoder an autoregressive model?

The normal Transformer decoder is autoregressive at inference time and non-autoregressive at training time. The non-autoregressive training can be done because of two factors: We don't use the ...
  • 16.3k
4 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
  • 1,426
4 votes

Preprocessing for Text Classification in Transformer Models (BERT variants)

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare ...
4 votes

Custom functions and pipelines

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself: [INIT] Are there any parameters in my logic? The variable you want to impute and the ...
  • 724
4 votes
Accepted

Can BERT be used for predicting words?

pip install transformers Then try this ...
4 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
4 votes
Accepted

Unigram tokenizer: how does it work?

The explanation in the documentation of the Huggingface Transformers library seems more approachable: Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving ...
  • 16.3k
4 votes
Accepted

What are the inputs to the first decoder layer in a Transformer model during the training phase?

Following your example: The source sequence would be How are you ...
  • 16.3k
4 votes
Accepted

How to use BERT in seq2seq model?

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain ...
  • 16.3k

Only top scored, non community-wiki answers of a minimum length are eligible