Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
68
votes
4answers
52k views

What is the positional encoding in the transformer model?

I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know what positional encoding is. by listening to some youtube videos I've found out that ...
17
votes
4answers
3k views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
10
votes
2answers
6k views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a Word2Vec or Glove,...
3
votes
1answer
630 views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
0
votes
1answer
204 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
10
votes
1answer
7k views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
17
votes
1answer
8k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
5
votes
3answers
3k views

Is time series forecasting possible with a transformer?

For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then ...
2
votes
1answer
377 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
2
votes
1answer
142 views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
5
votes
1answer
449 views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
3
votes
1answer
969 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
3
votes
1answer
67 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
3
votes
0answers
75 views

VQ-GAN understanding

I tried to understand how VQ-GAN works, but unfortunately I have not understood it. I tried to read some articles about it and watch a video. I believe a good and simple article will help me. You ...
2
votes
3answers
1k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
1answer
114 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
0
votes
1answer
18 views

An issue for sub-word tokenization preprocessing transformer

I'm stacked with executing the sub-word tokenization preprocessing to use transformer. According to the tutorial on the article, I have executed the sample code. However, one function was not defined ...
0
votes
1answer
973 views

Which is better: GPT or RelGAN for text generation?

Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN. So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN I am so ...
-2
votes
1answer
73 views

Transformers understanding

i have i a big trouble. I don't understand transformers. I understand embedding, rnn's, GAN's, even Attention. But i don't understand transformers. Approximately 2 months ago i decided to avoid usage ...