38

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...


14

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of the masked ...


8

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...


5

Here's the list of difference that I know about attention (AT) and self-attention (SA). In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - ...


3

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


3

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


3

pip install transformers Then try this from transformers import pipeline nlp = pipeline("fill-mask", model="bert-base") nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


3

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


3

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


3

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...


2

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target sequence length). The Query is what determines the output sequence length, therefore we obtain a sequence of the correct length (i.e. target sequence length). ...


2

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution $$P = softmax(\{M(X_1), M(X_2)\})$$ and take the state with the highest probability $$X=argmax_{X_i}(P)$$ But what if you actually want to sample from $P$ ...


2

In self-attention, it is not the decoder attending the encoder, but the layer attends itself, i.e., the queries and values are the same. In practice, this is usually done in the multi-head setup. You can view that as every head focusing on collecting different kinds of information from the hidden states. In multi-headed attention with $H$ heads, you first ...


2

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you ...


1

So the question is concerned about understanding the self-attention mechanism in greater detail, in particular how this idea of multi-head self-attention is used to compute strength of relations between tokens. I think it's best you look through this great tutorial on self-attention and see if this helps in your understanding of multi-head self-attention: ...


1

So the question asks between the difference between an attention vector and a positional vector. To answer this question, will give some context into how the transformer differs from a sequential model, such as RNNs and LSTMs. In the case of RNNs and LSTMs, data is fed sequentially "one-by-one" into the model to predict the output (whether that is ...


1

First to equation (7): $s_{i-1}$ is a vector, not a matrix. When you multiply it with a matrix $W$, you get another vector of which as the length of the intermediate attention projection, let us call it $d_a$. The shape of $W$ is thus $1024 \times d_a$. Similarly, the shape of $V$ is $512 \times d_a$ and bias $b$ is a vector of length $d_a$. The vector $w$ ...


1

In the simplest case, doing regression with Transformers is just a matter of changing the loss function. BERT-like models that use the representation of the first technical token as an input to the classifier. You can replace the classifier with a regressor and pretty much nothing will change. The error from the regressor will get propagated to the rest of ...


1

The general formulation of attention with queries, keys and values corresponds to a re-retrieval view on attention: you have some queries that you use to retrieve some values based on keys that correspond to them. With RNNs, attention is used for sequence-to-sequence models like machine translation. (Time series forecasting is usually formulated as sequence ...


1

Q, K, V vectors are trained with standard backpropagation. All trainable parameters are initialized at random, and then adjusted step by step with a Gradient Descent algorithm. Surprisingly, they are trained just as any standard ANN! It's pretty amazing what they can achieve with such a classical trick.


1

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...


1

Let's take the common translation task which transformers can be used for as an example: If you would like to translate English to German one example of your training data could be ("the cat is black", "die Katze ist schwarz"). In this case your target is simply the German sentence "die Katze ist schwarz" (which is of course not processed as a string but ...


1

Attention weights are learned through backpropagation, just like canonical layer weights. The hard part about attention models is to learn how the math underlying alignment works. Different formulations of attention compute alignment scores in different ways. The main is Bahdanau attention, formulated here. The other is Luong's, provided in several variants ...


1

Are you reffering to biological "cognitive function attention"? If yes than it could be argued its different. TL;DR Fixed-length Context vector in the encoder decoder architecture could not remember long term dependencies. Attention could learn these long term dependencies, hence have a long-term memory. If we speak about human attention it is ...


1

Here's the Deeplearning.ai notebook that is going to be helpful to understand it. Neural machine translation with attention


1

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions. On the other hand paper on Convolutional Sequence to Sequence ...


1

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature? You don't necessarily need Gumbel-Softmax to obtain "one-hot like" vectors, or the ability to differentiate through an indexing mechanism. The LSTM architecture and derived variants ...


1

For the softmax function, no matter what is the temperature, it is not the exact one-hot vector. If you could accept a soft version, it is good. However, if you choose the argmax to be the one, it is non-differentiable. One alternative way to back-propagate the gradients is by using the Straight Through Estimator (STE)[1] trick, and directly back-propagate ...


1

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We ...


Only top voted, non community-wiki answers of a minimum length are eligible