Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

116 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
4
votes
0answers
160 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
4
votes
1answer
534 views

What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
3
votes
0answers
86 views

ResNet50 + Transformer

In many papers people extract features from image using ResNet and than pass them through transformer. I want to implement the same. I want to get features and than classify them using transformer. ...
3
votes
0answers
75 views

VQ-GAN understanding

I tried to understand how VQ-GAN works, but unfortunately I have not understood it. I tried to read some articles about it and watch a video. I believe a good and simple article will help me. You ...
3
votes
0answers
77 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
3
votes
1answer
417 views

Why is 10000 used as the denominator in Positional Encodings in the Transformer Model?

I was working through the All you need is Attention paper, and while the motivation of positional encodings makes sense and the other stackexchange answers filled me in on the motivations of the ...
3
votes
1answer
775 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
2
votes
0answers
9 views

Best practices to train a transformer text classifier to predict/handle unseen labels

I fine-tuned a RoBERTa sequence classifier to classify paragraphs of certain documents using labeled paragraphs only (and skipping paragraphs with no label given). The model was validated and tested ...
2
votes
1answer
45 views

Why do Transformers need positional encodings?

At least in the first self-attention layer in the encoder, inputs have a correspondence with outputs, I have the following questions. Isn't ordering already implicitly captured by the query vectors, ...
2
votes
1answer
372 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
2
votes
0answers
58 views

How to use paraphrase_mining using sentence transformers pre-trained model

I am trying to find similarity between sentences using a pre-trained sentence-transformers model. I am trying to follow the code here - https://www.sbert.net/docs/usage/paraphrase_mining.html In trial ...
2
votes
0answers
140 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
2
votes
3answers
1k views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
2
votes
0answers
362 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
0answers
35 views

Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?

As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch. I was thinking of using an existing Word2Vec embedding ...
2
votes
0answers
240 views

Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all ...
2
votes
0answers
41 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
1
vote
0answers
19 views

Model doesn't know German well enough

I have a model that generates questions and answers based on input text. The texts are in German and based on observations it seems like the model doesn't know German well enough. I need to pretrain ...
1
vote
0answers
12 views

Any transformer model (NLP) for code classification?

Does any Transformer (NLP) that is suitable for code classification tasks exist? For example, I have a lot of source codes of various categories (driver, game, email client, etc.). I want to ...
1
vote
0answers
40 views

How to Fine Tune a BERT model for sentiment analysis to get the best f1 score

I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score. More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
1
vote
1answer
50 views

What does "expansion layer" mean?

Recently, I found "expansion layer" term in the next paper: Liu, Ze, et al. "Swin transformer: Hierarchical vision transformer using shifted windows." arXiv preprint arXiv:2103....
1
vote
0answers
12 views

Machine translation transformer output - "unknown" tokens?

Cross post from my original post in Stackoverflow: https://stackoverflow.com/questions/69595863/machine-translation-transformer-output-unknown-tokens Based on the feedback , I have now updated my ...
1
vote
0answers
55 views

NER prections with distilbert transformer model

I am trying to extract 'agreement date' label from a corpus of legal contracts. In the train dataset, I used pytorch-transformer model to train. ...
1
vote
0answers
16 views

Can a reformer model really handle long-range dependency?

I read this article about new attention model called Reformer. Here is the main strength of this model: The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
1
vote
0answers
11 views

What is the principial difference between zero-shot learning and k-NN and clusterization based methods?

One can consider clustering and k-NN to be a zero-shot, too? I think there is no much principal difference, except using some neural network architecture (usually it is a transformer) which is used to ...
1
vote
1answer
110 views

Not clear about relative position bias

I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
1
vote
1answer
57 views

How is attention different from linear MLPs?

Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
1
vote
0answers
75 views

Spatial positional encodings Vs Learned positional encodings(Object queries)

I have been trying to understand facebook's Detection transformer(DeTr) paper. Architecture Most of the explanation about the architecture is straightforward. I don't especially understand the ...
1
vote
0answers
20 views

Why do we need dot product as part of the Transformer's training process?

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
1
vote
1answer
196 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
1
vote
1answer
959 views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
1
vote
0answers
36 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...
1
vote
0answers
55 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...
1
vote
0answers
16 views

What does attention weights output from Transformer network do?

I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as ...
1
vote
0answers
184 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
1
vote
2answers
52 views

Can Transformer Models be used for Training Chatbots?

Can Transformer Models be used for Training Chatbots? Note - I am talking about the transformer model google released on the paper 'Attention is all you need'
1
vote
0answers
228 views

SVM on BERT-Embeddings with very small dataset does not converge

I am trying to reproduce the results from this paper where they use a linear SVM on top of BERT-Embeddings for text-classification. They use the average of the token-embeddings which results in a 768 ...
1
vote
0answers
307 views

What is the difference between register_buffer() and parameter.detach() in PyTorch?

I am writing a PositionalEmbedding() module which is an implementation based on "Attention Is All You Need" using PyTorch. According to the paper, there ...
1
vote
0answers
93 views

Calculating key and value vector in the Transformer's decoder block

I am implementing the transformer model in Pytorch by following Jay Alammar's post and the implementation here. My question is regarding the input to the decoder layer. As shown in the diagram above, ...
1
vote
1answer
96 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
1
vote
0answers
40 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
1
vote
1answer
176 views

How do I implement Dual-encoder model in Pytorch?

I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model. Here the encoder and decoder share the same weights but I am unable to ...
1
vote
0answers
235 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
0answers
64 views

How does Byte Pair Encoding work on the byte sequence?

I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant ...
0
votes
0answers
9 views

How can BERT be applied in image captioning?

1- How can BERT be applied in image captioning ? 2- Does the work on BERT is text classification in image captioning? 3- Do I need to get embedding for sentences or word as BERT work with embedding in ...
0
votes
1answer
9 views

Get sentence embeddings of transformer-based models

I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...). I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
0
votes
2answers
19 views

How to add positional embedding in an attention based model which takes custom feature embedding as input?

I am working on building an attention based deep neural network model where I fed this with a variable length of custom feature embeddings. I would like to provide positional information for my custom ...
0
votes
0answers
19 views

Heatmap on visual transformer patches

I have a vector with size [1, n_patches], each values is a score between 0 to 1 represent the relevancy of each patch - I want to create heatmap visualization on ...
0
votes
1answer
32 views

Variational Autoencoders VS Transformers

I'm relatively new to the field, but I'd like to know how do variational autoencoders fare compared to transformers?
0
votes
0answers
15 views

Algorithms for classification of very short text

I am to create a classification model for texts that typically have 3 to 4 words in them. I thought of using BERT and XLNet but not sure if they are the right choice for texts that short. Are there ...