Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
0
votes
1answer
12 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
0
votes
0answers
10 views

How to determine sentence similarity labels for sentence transformer fine-tuning?

I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...
0
votes
1answer
18 views

why do transformers mask at every layer instead of just at the input layer?

working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one ...
1
vote
0answers
19 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
-1
votes
0answers
11 views

Text generation based on properties/content, not previous words

Generators based on the transformer architecture predict the next token based on given previous token. So you can give "Today is" as input to a transformer and it will create a sentence or ...
5
votes
2answers
122 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
0answers
13 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
1
vote
0answers
25 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
0
votes
1answer
25 views

What is the difference between GPT blocks and BERT blocks

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though. If GPT uses Decoder only and BERT uses ...
0
votes
0answers
14 views

List of Google T5 possible operations

I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
0
votes
1answer
17 views

What is the number of neurons for the input layer of the BERT?

I think it is the vocab size. However I am not sure and I appreciate your help.
1
vote
0answers
12 views

What does attention weights output from Transformer network do?

I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as ...
2
votes
1answer
19 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...
2
votes
0answers
21 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
2
votes
2answers
28 views

Can Transformer Models be used for Training Chatbots?

Can Transformer Models be used for Training Chatbots? Note - I am talking about the transformer model google released on the paper 'Attention is all you need'
0
votes
0answers
19 views

How to use Pytorch's Transformer module “out of the box”

I am working on implementing my first transformer, and recently I've been working in Pytorch and I see that they offer a pre-packaged transformer model. Here are the docs. I have been reading through ...
0
votes
1answer
15 views

Many questions training unbalanced and duplicated data

I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems: ...
0
votes
0answers
17 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...
0
votes
1answer
48 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
-1
votes
1answer
50 views

What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
2
votes
1answer
29 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
1
vote
1answer
14 views

Understanding the XLNet model for a concrete case

I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it: Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (...
-1
votes
1answer
28 views

In transformers, do you understand why are the Value (V) vectors comes from the encoder? And than normalize with the query (Q) vector? [closed]

In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize. Can one please give some advise to the motivation of it? Or maybe ...
0
votes
1answer
28 views

Predict customer behaviour with Transformer(attention is all you need)

Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
3
votes
2answers
114 views

Effect of Stop-Word Removal on Transformers for Text Classification

The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect ...
0
votes
0answers
13 views

OneHotEncoder in ColumnTransformer: passthrough not returning original column names

I'm using a column transformer to pass in categorical data of the Kaggle Titanic dataset to one-hot encode. I'm using a pipeline as well so I can expand the process later on, but the dataframe column ...
0
votes
0answers
18 views

Where does the evaluation speed advantage of Transformer-XL come from?

The Transformer-XL paper claims an advantage in evaluation speed 363x-1874x than that of a baseline Transformer model. However, I do not understand where this massive difference comes from. Although ...
2
votes
1answer
128 views

What is the difference between BERT architecture and vanilla Transformer architecture

I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
0
votes
0answers
25 views

BERT data cleaning [duplicate]

I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data. Which steps should be performed? Does it make sense to perform a stemming or ...
0
votes
1answer
99 views

Why transform embedding dimension in sin-cos positional encoding?

Positional encoding using sine-cosine functions is often used in transformer models. Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the ...
1
vote
2answers
39 views

Role of decoder in Transformer?

I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
47 views

Why this TensorFlow Transformer model has Linear output instead of Softmax?

I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation. I am quite surprised that when the Transformer is created, their final output is a Dense ...
0
votes
1answer
63 views

What is the difference between GPT blocks and Transformer Decoder blocks?

I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same? In the original ...
1
vote
0answers
13 views

How to use paraphrase_mining using sentence transformers pre-trained model

I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html In trial ...
1
vote
1answer
41 views

Why does an attention layer in a transformer learn context?

I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers. What I'm confused on is why the ...
0
votes
2answers
33 views

Understanding Transformer's Self attention calculations

I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
0
votes
1answer
107 views

Weights shared by different parts of a transformer model

Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?
0
votes
0answers
17 views

Why do we use a d_dimentionap vector for positional encoding in a transformer?

It is mentioned in the paper attention is all you need that for any PE_t+k sinusoidal positional embedding can be calculated using a linear transformation of PE_t. But the same is true if PE is only ...
2
votes
1answer
22 views

How to train a model on top of a transformer to output a sequence?

I am using huggingface to build a model that is capable of identifying mistakes in a given sentence. Say I have a given sentence and a corresponding label as follows -> ...
1
vote
1answer
54 views

Can I fine-tune the BERT on a dissimilar/unrelated task?

In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned: "During pre-training, the model is trained on unlabeled data over different pre-training tasks." I am not sure if I ...
1
vote
0answers
36 views

SVM on BERT-Embeddings with very small dataset does not converge

I am trying to reproduce the results from this paper where they use a linear SVM on top of BERT-Embeddings for text-classification. They use the average of the token-embeddings which results in a 768 ...
0
votes
0answers
18 views

Question about Relative-Position-Representation code

In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py In _relative_attention_inner method, which I think is one of the ...
2
votes
0answers
31 views

Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
1
vote
1answer
143 views

What would be the target input for Transformer Decoder during test phase?

The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer I am having confusion about what ...
4
votes
1answer
176 views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
2
votes
2answers
39 views

How to treat data transformation choices as hyperparemeters?

While reading the book hands-on ML by Aurelien Geron, I came across this line- Treat your data transformation choices as hyperparameters, especially when you are not sure about them (e.g., if you’re ...
1
vote
1answer
76 views

Question about BERT embeddings with high cosine similarity

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
1
vote
2answers
71 views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
0
votes
0answers
13 views

Reformer the Efficient Transformer and Image Size Limits

I'm currently trying to use the Reformer: Image Generator with my own dataset. The colab notebook for the model is here: https://colab.research.google.com/github/google/trax/blob/master/trax/models/...
2
votes
0answers
56 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...