Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
1
vote
0answers
184 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
1
vote
2answers
53 views

Can Transformer Models be used for Training Chatbots?

Can Transformer Models be used for Training Chatbots? Note - I am talking about the transformer model google released on the paper 'Attention is all you need'
0
votes
1answer
19 views

Many questions training unbalanced and duplicated data

I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems: ...
0
votes
1answer
817 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
0
votes
1answer
1k views

What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
3
votes
1answer
67 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
1
vote
1answer
16 views

Understanding the XLNet model for a concrete case

I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it: Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (...
-1
votes
1answer
53 views

In transformers, do you understand why are the Value (V) vectors comes from the encoder? And than normalize with the query (Q) vector? [closed]

In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize. Can one please give some advise to the motivation of it? Or maybe ...
0
votes
1answer
85 views

Predict customer behaviour with Transformer(attention is all you need)

Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
4
votes
2answers
773 views

Effect of Stop-Word Removal on Transformers for Text Classification

The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect ...
3
votes
1answer
879 views

What is the difference between BERT architecture and vanilla Transformer architecture

I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
0
votes
0answers
193 views

BERT data cleaning [duplicate]

I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data. Which steps should be performed? Does it make sense to perform a stemming or ...
2
votes
1answer
864 views

Why transform embedding dimension in sin-cos positional encoding?

Positional encoding using sine-cosine functions is often used in transformer models. Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the ...
2
votes
2answers
460 views

Role of decoder in Transformer?

I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
174 views

Why this TensorFlow Transformer model has Linear output instead of Softmax?

I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation. I am quite surprised that when the Transformer is created, their final output is a Dense ...
1
vote
1answer
1k views

What is the difference between GPT blocks and Transformer Decoder blocks?

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same? In the original ...
2
votes
0answers
58 views

How to use paraphrase_mining using sentence transformers pre-trained model

I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html In trial ...
1
vote
1answer
123 views

Why does an attention layer in a transformer learn context?

I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers. What I'm confused on is why the ...
0
votes
2answers
82 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
2
votes
1answer
1k views

Weights shared by different parts of a transformer model

Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?
2
votes
1answer
85 views

How to train a model on top of a transformer to output a sequence?

I am using huggingface to build a model that is capable of identifying mistakes in a given sentence. Say I have a given sentence and a corresponding label as follows -> ...
1
vote
1answer
258 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
0answers
230 views

SVM on BERT-Embeddings with very small dataset does not converge

I am trying to reproduce the results from this paper where they use a linear SVM on top of BERT-Embeddings for text-classification. They use the average of the token-embeddings which results in a 768 ...
3
votes
1answer
419 views

Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
1
vote
1answer
915 views

What would be the target input for Transformer Decoder during test phase?

The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer I am having confusion about what ...
4
votes
1answer
522 views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
2
votes
2answers
62 views

How to treat data transformation choices as hyperparemeters?

While reading the book hands-on ML by Aurelien Geron, I came across this line- Treat your data transformation choices as hyperparameters, especially when you are not sure about them (e.g., if you’re ...
1
vote
1answer
194 views

Question about BERT embeddings with high cosine similarity

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
1
vote
2answers
527 views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
2
votes
0answers
140 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
2
votes
1answer
239 views

Transformer masking during training or inference?

I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
0
votes
1answer
92 views

Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?

fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
1
vote
2answers
479 views

Splitting into multiple heads -- multihead self attention

So, I have a doubt in Attention is all you need: The implementation of transformers on tensorflow's official documentation says: Each multi-head attention block gets three inputs; Q (query), K (key), ...
0
votes
2answers
319 views

What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
1
vote
1answer
185 views

Based on transformer, how to improve the text generation results?

If I do not pretrain the text generation model like BART, how to improve the result based on transformer like tensor2tensor? What are the improvement ideas for transformer in text generation task?
1
vote
3answers
3k views

Overfitting while fine-tuning pre-trained transformer

Pretrained transformers (GPT2, Bert, XLNET) are popular and useful because of their transfer learning capabilities. Just as a reminder: The goal of Transfer learning is is to transfer knowledge gained ...
2
votes
1answer
101 views

Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow. I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline They implement the positional ...
0
votes
1answer
224 views

NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

I am new to NLP and I just finished reading the paper "Attention is all you need". I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
-1
votes
1answer
815 views

For NLP, is GPT-3 better than RoBERTa? [closed]

I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
0
votes
1answer
17 views

Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

My task is to generate keywords from sentences. I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens. Pretraining batch_size = 8 and step = 1000000 ...
0
votes
2answers
49 views

BERT reasoning capabilities

I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://...
1
vote
1answer
59 views

what is the difference between positional vector and attention vector used in transformer model?

what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
2
votes
3answers
1k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
4answers
1k views

Next sentence prediction in RoBERTa

I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments ...
1
vote
0answers
307 views

What is the difference between register_buffer() and parameter.detach() in PyTorch?

I am writing a PositionalEmbedding() module which is an implementation based on "Attention Is All You Need" using PyTorch. According to the paper, there ...
1
vote
0answers
93 views

Calculating key and value vector in the Transformer's decoder block

I am implementing the transformer model in Pytorch by following Jay Alammar's post and the implementation here. My question is regarding the input to the decoder layer. As shown in the diagram above, ...
0
votes
1answer
32 views

What are the simplest methods for the label noise problem?

If I have enough low quality data from unsupervised methods or rule-based methods. I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex ...
0
votes
1answer
114 views

How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
6
votes
2answers
2k views

BPE vs WordPiece Tokenization - when to use / which?

What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a ...
1
vote
2answers
382 views

TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

The input function code: ...