57

For example, for word $w$ at position $pos \in [0, L-1]$ in the input sequence $\boldsymbol{w}=(w_0,\cdots, w_{L-1})$, with 4-dimensional embedding $e_{w}$, and $d_{model}=4$, the operation would be $$\begin{align*}e_{w}' &= e_{w} + \left[sin\left(\frac{pos}{10000^{0}}\right), cos\left(\frac{pos}{10000^{0}}\right),sin\left(\frac{pos}{10000^{2/4}}\right),...


19

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of ...


17

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking excerpts from the video, let us try understanding the “sin” part of the formula to compute the position embeddings: Here “pos” refers to the position of the “...


14

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...


12

Positional encoding is a re-representation of the values of a word and its position in a sentence (given that is not the same to be at the beginning that at the end or middle). But you have to take into account that sentences could be of any length, so saying '"X" word is the third in the sentence' does not make sense if there are different length sentences:...


10

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks. the target tokens decoded up to the current decoding step: for the first step, the matrix contains in ...


10

This is specified in the original Transformer paper, at the end of section 3.4: Transcription: 3.4 Embeddings and Softmax Similarly to other sequence transduction models, we use learned embeddings to convert the input tokens and output tokens to vectors of dimension 𝑑model. We also use the usual learned linear transformation and softmax function to ...


8

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) also sometimes help (0.1-0.3 might be reasonable values). If you have many input classes, label smoothing can help. You can try a smaller model dimension. If you ...


7

I will take as reference fairseq's implementation of the Transformer model. With this assumption: In the transformer, masks are used for two purposes: Padding: in the multi-head attention, the padding tokens are explicitly ignored by masking them. This corresponds to parameter key_padding_mask. Self-attention causality: in the multi-head attention blocks ...


7

The restriction in the maximum length of the transformer input is due to the needed amount of memory to compute the self-attention over it. The amount of memory needed by the self-attention in the Transformer is quadratic on the length of the input. This means that increasing the maximum length of the input, increases drastically the needed memory for self-...


6

To add to other answers, OpenAI's ref implementation calculates it in natural log-space (to improve precision, I think. Not sure if they could have used log in base 2). They did not come up with the encoding. Here is the PE lookup table generation rewritten in C as a for-for loop: int d_model = 512, max_len = 5000; double pe[max_len][d_model]; for (int i = ...


6

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum. This limitation, nevertheless, is not arbitrary, but has a deeper ...


5

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


5

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


4

When you concatenate, you have to define a priori the size of each vector to be concatenated. This means that, if we were to concatenate the token embedding and the positional embedding, we would have to define two dimensionalities, $d_t$ for the token and $d_p$ for the position, with the total dimensionality $d = d_t + d_p$, so $d>d_t$ and $d>d_p$. We ...


4

If you read the mentioned answer, I guess you already have the notion of the need for a encoding way to represent the position of the word in the input. In order not to use a sequence of integers (1, 2, 3, ... n) because of the lack of boundary in the value and the magnitude, a float friendly is preferred. But, just using a limited (0 to 1) option means you ...


4

To include this logic into a pipeline you have to create a custom transformer. You need to ask yourself: [INIT] Are there any parameters in my logic? The variable you want to impute and the category you want this imputation to be based on. [FIT] What part of the logic is related to computing what the transformation will be? When you compute the median()...


4

pip install transformers Then try this from transformers import pipeline nlp = pipeline("fill-mask", model="bert-base") nlp(f"This is the best thing I've {nlp.tokenizer.mask_token} in my life.")


4

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


4

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder BERT ends with the final representation of the words after the encoder is done processing it. In Transformer, the above is used in the decoder. That piece of architecture is not there in BERT


4

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, by Neo Wu, Bradley Green, Xue Ben, & Shawn O'Banion The Time Series Transformer, by Theodoras Ntakouris ...


3

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous layers. In encoder-decoder attention, the queries are decoder states from the previous layer, keys and values and the encoder states. In Equation 1 of the ...


3

A quick experiment you can do is to once do the preprocessing steps that you usually do and then feed it to the model and get the results. And once feed the dataset as it is to the model to compare the difference. In my experience doing the preprocessing won't make any difference, based on the dataset it gave me 1 more or less percent difference in accuracy (...


3

FunctionTransformer is useful because it allows you to apply a custom function in a pipeline. Because Pipeline() from sklearn.pipeline only works with objects that implement the .transform() and .fit() methods, you use FunctionTransformer to change your custom function to allow .transform() and/or .fit() to be used on it. You could transform a DataFrame ...


3

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and replace the encoder with XLM-R. Another option would be using Huggingface's Transofrmers that also provides basic support for sequence-to-sequence models as ...


3

I'm not quite sure how's the decoder output is flattened into a single vector That's the thing. It isn't flattened into a single vector. The linear transformation is applied to all $M$ vectors in the sequence individually. These vectors have a fixed dimension, which is why it works.


3

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future", i.e., those that will not be known at the inference time, because they will not have been generated yet. At inference time, it is no longer a problem because ...


3

BERT-style finetuning is known for its instability. Some aspects to take into account when having this kind of issues are: The number of epochs typically used to finetune BERT models is normally around 3. The main source of instability is that the authors of the original BERT article suggested using the Adam optimizer but disabling the bias compensation (...


3

Thank-you!! I'd also missed that multiply in my (fairseq transformer) code study, and it helps clear up a mystery that I'd noted: the (sinusoidal, non-learned) positional embeddings are initialized with a range of -1.0 to +1.0, but the word-embeddings are initialized with a mean of 0.0 and s.d. of embedding_dim ** -0.5 (0.044 for 512, 0.03125 for 1024). So, ...


3

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality. Another option would be to reuse just the embeddings instead of the whole model.


Only top voted, non community-wiki answers of a minimum length are eligible