63 votes

What is the positional encoding in the transformer model?

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$...
  • 8,667
34 votes
Accepted

What is the positional encoding in the transformer model?

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking ...
  • 456
24 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Here's the list of difference that I know about attention (AT) and self-attention (SA). In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have ...
  • 341
21 votes
Accepted

Can BERT do the next-word-predict task?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you ...
  • 16.3k
17 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

Let me try to keep it more intuitive and less mathematical Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give ...
  • 683
16 votes
Accepted

Why is the decoder not a part of BERT architecture?

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
  • 16.3k
13 votes

Gumbel-Softmax trick vs Softmax with temperature

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution $$...
12 votes

What is the positional encoding in the transformer model?

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take ...
10 votes
Accepted

Transformer model: Why are word embeddings scaled before adding positional encodings?

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings ...
  • 16.3k
8 votes
Accepted

Variable input/output length for Transformer

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target ...
  • 16.3k
7 votes

Why does the transformer positional encoding use both sine and cosine?

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, ...
  • 1,772
6 votes

How does attention mechanism learn?

From the Amazing Blog - FloydHub Blog- Attention Mechanisms Attention Mechanisms Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the ...
  • 2,466
6 votes
Accepted

How do attention mechanisms in RNNs learn weights for a variable length input

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and ...
  • 8,667
6 votes

What is the positional encoding in the transformer model?

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the ...
  • 61
6 votes
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
  • 1,426
5 votes
Accepted

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have ...
  • 16.3k
5 votes
Accepted

Attention for time-series in neural networks

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the ...
  • 683
4 votes

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...
  • 1,426
4 votes
Accepted

Can BERT be used for predicting words?

pip install transformers Then try this ...
4 votes

Transformer model: Why are word embeddings scaled before adding positional encodings?

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized ...
3 votes

What is the advantage of positional encoding over one hot encoding in a transformer model?

You are mixing two different concepts in the same question: One hot encoding: approach to encode $n$ discrete tokens by having an $n$-dimensional vectors with all 0's except one 1. This can be used ...
  • 16.3k
3 votes

What is the advantage of positional encoding over one hot encoding in a transformer model?

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would ...
  • 1,426
3 votes

ValueError: Cannot convert a partially known TensorShape to a Tensor: (?, 256)

The problem is inside the sampling functions. I had the same problem and found out the answer in the tutorial here. my original code is: ...
3 votes

Gumbel-Softmax trick vs Softmax with temperature

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature? You don't necessarily ...
  • 131
3 votes
Accepted

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the ...
3 votes
Accepted

What reccent alternatives to LSTM are there for regression problems?

Transformers (aka "attention models") are being used in place of LSTMs in many areas, as they generally give better results, and/or are quicker to train. They can be used for regression ...
2 votes

What's the difference between Attention vs Self-Attention? What problems does each other solve that the other can't?

In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary ...
  • 121
2 votes

Gumbel-Softmax trick vs Softmax with temperature

For the softmax function, no matter what is the temperature, it is not the exact one-hot vector. If you could accept a soft version, it is good. However, if you choose the argmax to be the one, it is ...
  • 21
2 votes

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

the first few bits of the embedding are completely unusable by the network because the position encoding will distort them a lot This confused me very much at first because I was thinking of the ...
  • 129
2 votes

Why is the decoder not a part of BERT architecture?

BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for ...

Only top scored, non community-wiki answers of a minimum length are eligible