Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
40 views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
1
vote
1answer
23 views

Why are convolutions still used in some Transformer networks for speech enhancement?

So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...
2
votes
1answer
42 views

Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
0
votes
0answers
14 views

How does multi-head attention on “multiple attention axes” works?

I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a ...
0
votes
0answers
23 views

Save and Load Simple Transformer Model

I have trained Text classifier using simpleTranformer.ai I am struggling to save and load the model in docker container. Please let me know how can I save the trained model and then load it into ...
0
votes
2answers
27 views

Intuition of “Head” in Attention models (Transformer)?

I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!
0
votes
0answers
20 views

learn information from text and resolve problem using transformers

Let's imagine that we have some question, like this: "x multiplied by x equals 9. What is x?" For this easy question answer is +-3. I want to make AI model answer on questions like that. To ...
0
votes
0answers
12 views

Validation Loss Decreases Then Increases to The Same Values

I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing. ...
0
votes
0answers
9 views

Is the output of this loss function considered as a percentage or something else?

I am currently working on fine-tuning the PEGASUS model for abstractive summarization. The script for fine-tuning the class Trainer from Transformers is imported. The output of training loss is ...
0
votes
1answer
19 views

BERT Masked Language Model question

I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word ...
0
votes
0answers
20 views

Why do relative positional embeddings instead of absolute positional encoding improve the Transformer?

I've been researching the Music Transformer, the paper for which introduced an efficient algorithm to compute Relative Positional Embeddings in a Transformer. I know that Relative Positional ...
0
votes
0answers
105 views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
0
votes
1answer
11 views

There could be a problem with the linear layer after the attention inside a transformer?

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...
0
votes
1answer
25 views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...
1
vote
1answer
56 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...
0
votes
0answers
10 views

transformers require a large d_model even when the input cardinality is low?

I'm training a transformer encoder for an NLP task over character data, so the cardinality of my input is 26. I've noticed that if I want to create a strong model, I need make $x$ == my embedding ...
0
votes
1answer
105 views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
1
vote
1answer
33 views

Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...
0
votes
1answer
39 views

How to use BERT in seq2seq model? [closed]

I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...
0
votes
1answer
39 views

Are all 110 million parameter in bert are trainable

I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below? By trainable I understand they are ...
0
votes
1answer
49 views

What are the inputs to the first decoder layer in a Transformer model during the training phase?

I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in ...
1
vote
1answer
129 views

Backpropagation of a transformer

when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained ...
1
vote
1answer
117 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...
-2
votes
1answer
49 views

Transformers understanding

i have i a big trouble. I don't understand transformers. I understand embedding, rnn's, GAN's, even Attention. But i don't understand transformers. Approximately 2 months ago i decided to avoid usage ...
3
votes
2answers
104 views

Layer normalization details in GPT-2

I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works. Let's say that our ...
0
votes
1answer
18 views

Transformer architecture question

I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/. The first attention ...
1
vote
1answer
20 views

Train a final model with the full data

I have trained a few NLP models, measured their performances and now I want to create a final model for production trained with all the data I have available. I'm working in text classification and I'...
2
votes
1answer
52 views

How to evaluate the quality of speech-to-text data without access to the true labels?

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription ...
0
votes
1answer
38 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
0
votes
0answers
29 views

How to determine sentence similarity labels for sentence transformer fine-tuning?

I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...
0
votes
1answer
32 views

why do transformers mask at every layer instead of just at the input layer?

working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one ...
1
vote
0answers
24 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
7
votes
2answers
332 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
0answers
66 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
1
vote
0answers
40 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
0
votes
1answer
183 views

What is the difference between GPT blocks and BERT blocks

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though. If GPT uses Decoder only and BERT uses ...
0
votes
0answers
27 views

List of Google T5 possible operations

I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
0
votes
1answer
57 views

What is the number of neurons for the input layer of the BERT?

I think it is the vocab size. However I am not sure and I appreciate your help.
1
vote
0answers
12 views

What does attention weights output from Transformer network do?

I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as ...
1
vote
1answer
31 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...
1
vote
0answers
72 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
0
votes
2answers
34 views

Can Transformer Models be used for Training Chatbots?

Can Transformer Models be used for Training Chatbots? Note - I am talking about the transformer model google released on the paper 'Attention is all you need'
0
votes
0answers
25 views

How to use Pytorch's Transformer module “out of the box”

I am working on implementing my first transformer, and recently I've been working in Pytorch and I see that they offer a pre-packaged transformer model. Here are the docs. I have been reading through ...
0
votes
1answer
18 views

Many questions training unbalanced and duplicated data

I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems: ...
0
votes
0answers
26 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...
0
votes
1answer
221 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
-1
votes
1answer
263 views

What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
3
votes
1answer
38 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
1
vote
1answer
14 views

Understanding the XLNet model for a concrete case

I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it: Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (...
-1
votes
1answer
31 views

In transformers, do you understand why are the Value (V) vectors comes from the encoder? And than normalize with the query (Q) vector? [closed]

In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize. Can one please give some advise to the motivation of it? Or maybe ...