56

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...


19

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of ...


17

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking excerpts from the video, let us try understanding the “sin” part of the formula to compute the position embeddings: Here “pos” refers to the position of the “...


14

Here's the list of difference that I know about attention (AT) and self-attention (SA). In neural networks you have inputs before layers, activations (outputs) of the layers and in RNN you have states of the layers. If AT is used at some layer - the attention looks to (i.e. takes input from) the activations or states of some other layer. If SA is applied - ...


12

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...


11

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


11

Let's say you have two states, $X_1$ and $X_2$, and you have a model, $M$, that produces a score $M(X_i)$ for each state (i.e, the logits). Next you can use the logits to compute some distribution $$P = softmax(\{M(X_1), M(X_2)\})$$ and take the state with the highest probability $$X=argmax_{X_i}(P)$$ But what if you actually want to sample from $P$ ...


10

Let me try to keep it more intuitive and less mathematical Prior to 2014, RNNs used to perform badly if the sequence was beyond a certain size. After all RNNs encode all steps in the sequence and give out a final output which is 'supposed' to be something of a sequence embedding. This works well for short sequences but beyond a certain length, it starts '...


10

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension 𝑑model. We also use the usual learned linear transformation and softmax function to ...


6

From the Amazing Blog - FloydHub Blog- Attention Mechanisms Attention Mechanisms Attention takes two sentences, turns them into a matrix where the words of one sentence form the columns, and the words of another sentence form the rows, and then it makes matches, identifying relevant context. This is very useful in machine translation. When we think about ...


6

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


6

Your understanding is not correct: in the encoder-decoder attention, the Keys and Values come from the encoder (i.e. source sequence length) while the Query comes from the decoder itself (i.e. target sequence length). The Query is what determines the output sequence length, therefore we obtain a sequence of the correct length (i.e. target sequence length). ...


5

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


4

Attention weight $\boldsymbol{\alpha}$ is not, and need not to be, constrained in size. For source sequence $\boldsymbol{x} = x_1\cdots x_{T_x}$ (where $T_x$ can vary from one source to another) and target sequence $\boldsymbol{y} = y_1...y_{T_y}$ (where $T_y$ can also vary from one target to another), weight $\boldsymbol{\alpha}_i = (\alpha_{i1},\cdots,\...


4

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We ...


4

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you ...


4

pip install transformers Then try this from transformers import pipeline nlp = pipeline("fill-mask", model="bert-base") nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


4

It is an interesting question. I would not completely agree with you though when you say that most time-series models dont use attention. However there is not as much documentation available on the web as there is for other applications. LSTNet was one of the first papers that proposed using an LSTM + attention mechanism for multivariate forecasting time ...


3

From a practical and theoretical perspective, when is it beneficial to incorporate Gumbel noise into a neural network, as opposed to just using Softmax with temperature? You don't necessarily need Gumbel-Softmax to obtain "one-hot like" vectors, or the ability to differentiate through an indexing mechanism. The LSTM architecture and derived variants ...


3

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


3

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet. At inference time, it is no longer a problem because ...


3

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized with a range of -1.0 to +1.0, but the word-embeddings are initialized with a mean of 0.0 and s.d. of embedding_dim ** -0.5 (0.044 for 512, 0.03125 for 1024). So, ...


2

In attention mechanisms, you take an expectation of a representation of data V with respect to some probability mass function, thus computing the context vector, which is essentially a summary statistic (weighted mean) of your data: \begin{align} c&=\mathbb{E}_p[V] \end{align} The big question is how you determine the elements of $p$, the probability ...


2

For the softmax function, no matter what is the temperature, it is not the exact one-hot vector. If you could accept a soft version, it is good. However, if you choose the argmax to be the one, it is non-differentiable. One alternative way to back-propagate the gradients is by using the Straight Through Estimator (STE)[1] trick, and directly back-propagate ...


2

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would learn embeddings of earlier positions much more reliably than embeddings of later positions. On the other hand paper on Convolutional Sequence to Sequence ...


2

I've been also looking for the answer of this question, and I give my different view of Gumbel softmax just because I think this is a good question. From a general point of view: We use softmax normally because we need a so-called score, or a distribution $\pi_1 .. \pi_n$ for representing n probabilities of categorical variable with size n; We use Gumbel-...


2

BERT is a pretraining model to do the downstream tasks such as question answering, NLI and other language tasks. So it just needs to encode the language representations so that it could be used for other tasks.That's why it consists only of encoder parts. You can add the decoder while doing your specific task and this decoder could be anything based on your ...


2

In self-attention, it is not the decoder attending the encoder, but the layer attends itself, i.e., the queries and values are the same. In practice, this is usually done in the multi-head setup. You can view that as every head focusing on collecting different kinds of information from the hidden states. In multi-headed attention with $H$ heads, you first ...


2

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...


2

To answer your specific questions: AdditiveAttention() and Attention() layers, are (loosely but not exactly) based on Bahdanau and Luong's attentions, respectively. They use post-2018 semantics of queries, values and keys. To map the semantics to the Bahdanau or Luong's paper, you can consider the 'query' to be the last decoder hidden state. The 'values' ...


Only top voted, non community-wiki answers of a minimum length are eligible