Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
136
questions
0
votes
1answer
12 views
How do the linear layers in the attention mechanism work?
I think I now the answer to my question but I dont really get confirmation.
When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...
0
votes
0answers
10 views
How to determine sentence similarity labels for sentence transformer fine-tuning?
I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...
0
votes
1answer
18 views
why do transformers mask at every layer instead of just at the input layer?
working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one ...
1
vote
0answers
19 views
how is the linear relation between positional encoding helping attention?
I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
-1
votes
0answers
11 views
Text generation based on properties/content, not previous words
Generators based on the transformer architecture predict the next token based on given previous token. So you can give "Today is" as input to a transformer and it will create a sentence or ...
5
votes
2answers
122 views
Transformer model: Why are word embeddings scaled before adding positional encodings?
While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...
1
vote
0answers
13 views
How to i get word embeddings for out of vocabulary words using a transformer model?
When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
1
vote
0answers
25 views
Using numpy.ndarray in machine learning sklearn.preprocessing model
I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing:
I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
0
votes
1answer
25 views
What is the difference between GPT blocks and BERT blocks
Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though.
If GPT uses Decoder only and BERT uses ...
0
votes
0answers
14 views
List of Google T5 possible operations
I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
0
votes
1answer
17 views
What is the number of neurons for the input layer of the BERT?
I think it is the vocab size. However I am not sure and I appreciate your help.
1
vote
0answers
12 views
What does attention weights output from Transformer network do?
I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as ...
2
votes
1answer
19 views
Inference order in BERT masking task
In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order?
For example:
The ...
2
votes
0answers
21 views
How do I handle class imbalance for text data when using pretrained models like BERT?
I have a skewed dataset consisting of samples of the form:
Category 1 10000
Category 2 2000
Category 3 400
Category 4 300
Category 5 100
The dataset ...
2
votes
2answers
28 views
Can Transformer Models be used for Training Chatbots?
Can Transformer Models be used for Training Chatbots?
Note - I am talking about the transformer model google released on the paper 'Attention is all you need'
0
votes
0answers
19 views
How to use Pytorch's Transformer module “out of the box”
I am working on implementing my first transformer, and recently I've been working in Pytorch and I see that they offer a pre-packaged transformer model. Here are the docs. I have been reading through ...
0
votes
1answer
15 views
Many questions training unbalanced and duplicated data
I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems:
...
0
votes
0answers
17 views
What's the best method to merge N categorical features into one and keep it as categorical
I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels.
I have ...
0
votes
1answer
48 views
BERT uses WordPiece, RoBERTa uses BPE
In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned:
The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...
-1
votes
1answer
50 views
What's the right input for gpt-2 in NLP
I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...
2
votes
1answer
29 views
Trained BERT models perform unpredictably on test set
We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...
1
vote
1answer
14 views
Understanding the XLNet model for a concrete case
I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it:
Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (...
-1
votes
1answer
28 views
In transformers, do you understand why are the Value (V) vectors comes from the encoder? And than normalize with the query (Q) vector? [closed]
In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize.
Can one please give some advise to the motivation of it? Or maybe ...
0
votes
1answer
28 views
Predict customer behaviour with Transformer(attention is all you need)
Please advice, am I thinking correctly: is it possible to represent customer behavior data from an online store as a sequence data? Because it is describing interactions of the customer with the shop ...
3
votes
2answers
114 views
Effect of Stop-Word Removal on Transformers for Text Classification
The domain here is essentially topic classification, so not necessarily a problem where stop-words have an impact on the analysis (as opposed to, say, sentiment analysis where structure can affect ...
0
votes
0answers
13 views
OneHotEncoder in ColumnTransformer: passthrough not returning original column names
I'm using a column transformer to pass in categorical data of the Kaggle Titanic dataset to one-hot encode. I'm using a pipeline as well so I can expand the process later on, but the dataframe column ...
0
votes
0answers
18 views
Where does the evaluation speed advantage of Transformer-XL come from?
The Transformer-XL paper claims an advantage in evaluation speed 363x-1874x than that of a baseline Transformer model.
However, I do not understand where this massive difference comes from.
Although ...
2
votes
1answer
128 views
What is the difference between BERT architecture and vanilla Transformer architecture
I'm doing some research for the summarization task and found out BERT is derived from the Transformer model. In every blog about BERT that I have read, they focus on explaining what is a bidirectional ...
0
votes
0answers
25 views
BERT data cleaning [duplicate]
I am wondering which data cleaning steps should be performed if you want to re-fine a BERT model on custom text data.
Which steps should be performed?
Does it make sense to perform a stemming or ...
0
votes
1answer
99 views
Why transform embedding dimension in sin-cos positional encoding?
Positional encoding using sine-cosine functions is often used in transformer models.
Assume that $X \in R^{l\times d}$ is the embedding of an example, where $l$ is the sequence length and $d$ is the ...
1
vote
2answers
39 views
Role of decoder in Transformer?
I understand the mechanics of Encoder-Decoder architecture used in the Attention Is All You Need paper. My question is more high level about the role of the decoder. Say we have a sentence translation ...
0
votes
1answer
47 views
Why this TensorFlow Transformer model has Linear output instead of Softmax?
I am checking this official TensorFlow tutorial on a Transformer model for Portuguese-English translation.
I am quite surprised that when the Transformer is created, their final output is a Dense ...
0
votes
1answer
63 views
What is the difference between GPT blocks and Transformer Decoder blocks?
I know GPT is a Transformer-based Neural Network, composed of several blocks. These blocks are based on the original Transformer's Decoder blocks, but are they exactly the same?
In the original ...
1
vote
0answers
13 views
How to use paraphrase_mining using sentence transformers pre-trained model
I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html
In trial ...
1
vote
1answer
41 views
Why does an attention layer in a transformer learn context?
I understand the transformer architecture (from "Attention is All You Need"), as well as how the attention is computed in the multi-headed attention layers.
What I'm confused on is why the ...
0
votes
2answers
33 views
Understanding Transformer's Self attention calculations
I was going through this link: https://www.analyticsvidhya.com/blog/2019/06/understanding-transformers-nlp-state-of-the-art-models/?utm_source=blog&utm_medium=demystifying-bert-groundbreaking-nlp-...
0
votes
1answer
107 views
Weights shared by different parts of a transformer model
Which parts of a transformer share weights, like, do all the encoders share the same weight or all the decoders share the same weights?
0
votes
0answers
17 views
Why do we use a d_dimentionap vector for positional encoding in a transformer?
It is mentioned in the paper attention is all you need that for any PE_t+k sinusoidal positional embedding can be calculated using a linear transformation of PE_t. But the same is true if PE is only ...
2
votes
1answer
22 views
How to train a model on top of a transformer to output a sequence?
I am using huggingface to build a model that is capable of identifying mistakes in a given sentence.
Say I have a given sentence and a corresponding label as follows ->
...
1
vote
1answer
54 views
Can I fine-tune the BERT on a dissimilar/unrelated task?
In the original BERT paper, section 3 (arXiv:1810.04805) it is mentioned:
"During pre-training, the model is trained on unlabeled data over different pre-training tasks."
I am not sure if I ...
1
vote
0answers
36 views
SVM on BERT-Embeddings with very small dataset does not converge
I am trying to reproduce the results from this paper where they use a linear SVM on top of BERT-Embeddings for text-classification. They use the average of the token-embeddings which results in a 768 ...
0
votes
0answers
18 views
Question about Relative-Position-Representation code
In https://github.com/tensorflow/tensor2tensor/blob/master/tensor2tensor/layers/common_attention.py
In _relative_attention_inner method, which I think is one of the ...
2
votes
0answers
31 views
Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?
I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
1
vote
1answer
143 views
What would be the target input for Transformer Decoder during test phase?
The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer
I am having confusion about what ...
4
votes
1answer
176 views
Bert for QuestionAnswering input exceeds 512
I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
2
votes
2answers
39 views
How to treat data transformation choices as hyperparemeters?
While reading the book hands-on ML by Aurelien Geron, I came across this line-
Treat your data transformation choices as hyperparameters, especially
when you are not sure about them (e.g., if youāre ...
1
vote
1answer
76 views
Question about BERT embeddings with high cosine similarity
Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
1
vote
2answers
71 views
Does finetuning BERT involving updating all of the parameters or just the final classification layer?
Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
0
votes
0answers
13 views
Reformer the Efficient Transformer and Image Size Limits
I'm currently trying to use the Reformer: Image Generator with my own dataset. The colab notebook for the model is here:
https://colab.research.google.com/github/google/trax/blob/master/trax/models/...
2
votes
0answers
56 views
Loss first decreases and then increases
I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
