17

Here is an awesome recent Youtube video that covers position embeddings in great depth, with beautiful animations: Visual Guide to Transformer Neural Networks - (Part 1) Position Embeddings Taking excerpts from the video, let us try understanding the “sin” part of the formula to compute the position embeddings: Here “pos” refers to the position of the “...


5

Transformers can be used for time series forecasting. See the following articles: Adversarial Sparse Transformer for Time Series Forecasting, by Sifan Wu et al. Deep Transformer Models for Time Series Forecasting: The Influenza Prevalence Case, by Neo Wu, Bradley Green, Xue Ben, & Shawn O'Banion The Time Series Transformer, by Theodoras Ntakouris ...


5

(This answer was originally a comment) You can find the algorithmic difference here. In practical terms, their main difference is that BPE places the @@ at the end of tokens while wordpieces place the ## at the beginning. The main performance difference usually comes not from the algorithm, but the specific implementation, e.g. sentencepiece offers a very ...


4

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-based decoders are autoregressive by nature, Transformers are not. Instead, all predictions are generated at once based on the real target tokens (i.e. teacher ...


3

Look at https://ai.facebook.com/blog/using-neural-networks-to-solve-advanced-mathematics-equations/ Treat to your problem like to MT task. Use transformer.


3

In principle, it is possible to reuse the special tokens as you describe. However, according to research, you should not freeze BERT, but fine-tune the whole model with your data, in order to obtain better translation quality. Another option would be to reuse just the embeddings instead of the whole model.


2

You don't need to make preprocessing as I understand, and the reason for this is that the Transformer makes an internal "dynamic" embedding of words that are not the same for every word; instead, the coordinates change depending on the sentence being tokenized due to the positional encoding it makes. Note the difference with Word2Vec, GloVe or ...


2

The explanation in the documentation of the Huggingface Transformers library seems more approachable: Unigram is a subword tokenization algorithm introduced in Subword Regularization: Improving Neural Network Translation Models with Multiple Subword Candidates (Kudo, 2018). In contrast to BPE or WordPiece, Unigram initializes its base vocabulary to a large ...


2

Following your example: The source sequence would be How are you <EOS> The input to the encoder would be How are you <EOS>. Note that there is no <start> token here. The target sequence would be I am fine <EOS> . The output of the decoder will be compared against this in the training. The input to the decoder would be <start> I ...


2

The Transformer is a seq2seq model. At training time, you pass to the Transformer model both the source and target tokens, just like what you do with LSTMs or GRUs with teacher forcing, which is the default way of training them. Note that, in the Transformer decoder, we need to apply masking to avoid the predictions depending on the current and future tokens....


2

We find some justifications in the Conformer paper: Convolutions are better than Transformers at detecting fine-grained patterns: While Transformers are good at modeling long-range global context, they are less capable to extract fine-grained local feature patterns. Convolution neural networks (CNNs), on the other hand, exploit local information and are ...


2

I'm unclear whether transformers are the best tool for time series forecasting. Transformers at the end of the day are just the latest in a series of sequence-to-sequence models with an encoder and decoder. This means that transformers change something to something else. With time series you aren't changing something to something else, you're trying to find ...


2

Transformers were originally architected for NLP. However many studies have shown that they CAN be used for time series as well and with great success. Let us look at the differences and similarities and the industry developments around that particular area. Once the above points are understood, refining your transformer model to make it work for time-series ...


2

The last step for production models is typically to train with the entire set (train + validation) after tuning the hyperparameters using the validation set(s). The difference is typically not too drastic as the validation set should only be a fraction of the dataset but more data is always helpful especially for DL-based models. I'm not familiar with the ...


2

About the need for tgt_key_padding_mask While padding is usually applied after the normal tokens (i.e. right padding), it is perfectly fine to apply it before normal tokens (i.e. left padding). For instance, fairseq supports parameter left_pad to specify precisely this. For left padding to be handled correctly, you must mask the padding tokens, because the ...


2

$d_k$ is the dimensionality of the query/key/value vectors. In your example, the length of those vectors is 3, so $d_k = 3$


2

The masked language model task is the key to BERT and RoBERTa. However, they differ in how they prepare such masking. The original RoBERTa article explains it in section 4.1: BERT relies on randomly masking and predicting tokens. The original BERT implementation performed masking once during data preprocessing, resulting in a single static mask. To avoid ...


2

Your understanding is correct: in the encoder-decoder attention blocks, the Keys and Values are the output of the encoder, while the Query vectors come from the decoder layers. At inference time we have as many Query positions as the step we are in. Remember that at inference time, the decoder behaves autoregressive, meaning that at each timestep T it ...


2

An engineering solution would be: Create a language detector, feed the input to the detector, based on the language type classification, send the input to the appropriate model, that is if the language is french, feed the input directly to the CamemBERT. The output will be as accurate as the CamemBERT multiplied by the accuracy of the language detector. But ...


2

To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only model trained using the left-to-right language objective and operates autoregressively. Other than that, there are only technical differences in hyper-parameters, ...


1

All heads are fed the exact same input. Each head learns different weight values because: The attention heads, along with the rest of the network, are initialized randomly. The back-propagated gradients each head receives are different. This is because the result of the multi-head attention is the concatenation of each head. When back-propagating through ...


1

I think, what you are looking for is $d_k = d_v = d_{model}/h$ [1] where $h$ number of heads and $d_{model}$ dimensions of keys, values and queries for single attention version of the model. In relation of the model architecture and its embedding specifically, the above translates to $Query Size = Embedding Size / h$ Model input According to the above, at ...


1

In the original Transformer article, these linear layers are just matrix multiplications. As described in the paragraph you referred to in your question $W^Q$ is a matrix of dimensions $d_{model} \times d_k$, that is, it is a fully connected layer with $d_k$ units. In practical implementations, these have the optional addition of a bias vector. You can see ...


1

d_model is the dimensionality of the representations used as input to the multi-head attention, which is the same as the dimensionality of the output. In the case of normal transformers, d_model is the same size as the embedding size (i.e. 512). This naming convention comes from the original Transformer paper. depth is d_model divided by the number of ...


1

First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. Datasets (benchmarks) they used to train and evaluate this model from scratch were WMT 2014 Engligh-to-German, WMT 2014 English-to-French (section 5.1 of the paper)...


1

No, this is not a problem. If we zoom into the scaled dot product attention blocks, which happen before the projection with $W^O$ we see this: There, you can see how the masking of the current and future positions happens inside the scaled dot product attention, which happens before the multiplication by $W^O$. Therefore, the values learned for $W^O$ are ...


1

Self-attention means X pays attention to X, as opposed to "normal" attention where X pays attention to Y. Multi-head attention is as opposed to single-head attention. You can choose to use multi- or single-head attention equally for self-attention and for normal-attention. Masking X and/or Y is a third independent aspect of a design. In a ...


1

I like to think about it in the context of progression of attention mechanisms in neural networks. Early attention mechanisms were implemented as explicit sliding windows over the encoding sequence. For example, Graves 2013 implemented it as the average of several sliding Gaussians. So, it was a local attention mechanism to attend to a learnable window ...


1

There are some problems with your description: During training, the decoder receives all the shifted target tokens, prepending the BOS token. You removed sole. The actual input would be: [<bos>, io, amo, il, sole]. Note that the output at the position of sole would be the end-of-sequence token <eos>. During training, there is a single forward ...


1

All these parameters are trainable. Note that in normal Transformers it is typical to have fixed (non-trainable) positional embeddings, but in BERT they are learned. Note also the "pooler" component, which is an extra projection that was not mentioned in the paper, but which the authors commented on later.


Only top voted, non community-wiki answers of a minimum length are eligible