Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
290
questions
0
votes
0
answers
10
views
How to optimize hyperparameters in Bert?
I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
0
votes
0
answers
30
views
Can i use Transformer-XL for text classification task?
I want to use transformer xl for text classification tasks. But I don't know the architect model for the text classification task. I use dense layers with activation softmax for logits output from the ...
1
vote
1
answer
22
views
BERT base uncased required gpu ram
I'm working on an NLP task, using BERT, and I have a little doubt about GPU memory.
I already made a model (using DistilBERT) since I had out-of-memory problems with tensorflow on a RTX3090 (24gb gpu'...
0
votes
0
answers
7
views
BertTokenizer on custom data returns same index for all tokens
I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
0
votes
1
answer
11
views
In what ways are transformers better for Vision than CNNs?
I am looking at getting an intuition to how transformers can be better in vision tasks.
0
votes
0
answers
12
views
Model stalls and learning slows down after the first epochs
I have an issue with a model that I'm working on, I cannot show you the model architecture because it's basically confidential research.
The model includes Graph convolutional networks and Transformer ...
0
votes
0
answers
34
views
Transformer time series classification using time2vec positional embedding
I want to use a transformer model to do classification of fixed-length time series. I was following along this tutorial using keras which uses time2vec as a positional embedding. According to the ...
0
votes
0
answers
12
views
Creating a new prediction for the Keras.io BST model
The Keras.io example of a Transformer-based recommendation system is a great example for me to understanding neural networks in Keras. But how would you use the create_model_inputs() to get a new ...
0
votes
1
answer
16
views
How do i generate text from ids in Torchtext's sentencepiece_numericalizer?
The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
0
votes
0
answers
18
views
Can I use MLM method to fine tune my BERT model, if it was initially trained with natural language inference method?
I am using BERT model for sentence similarity task. However my dataset with sentence is very specific and I want to fine tune my model on it first. My dataset is unlabelled. And BERT model that I want ...
0
votes
0
answers
23
views
Should I pretrain my BERT model on specific dataset if it has only one class of labels?
I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
0
votes
0
answers
15
views
While training BERT variant, getting IndexError: index out of range in self
While training XLMRobertaForSequenceClassification:
...
1
vote
0
answers
15
views
Guide to Natural language Prompt programming for few-shot learning of Pretrained Language Models
I'm currently working on a project with the goal of producing AI content in the space of a content generation like blog writing, Instagram caption generation etc. Found the in-context few-shot ...
0
votes
1
answer
25
views
Why can't positions in transformers be simply appended to the input to preserve the positional information instead of using positional encodings?
I saw in an intro to transformers in this video that positional encodings need to be used to preserve positional information, otherwise word order may not be understood by the neural network. They ...
0
votes
0
answers
20
views
While doing inference with a Transformer-Decoder in batches, how can I stop each sequence separately (if possible)?
So my decoder is a transformer-decoder and in training I don't have any issue. I have all the input from the beggining and correctly masked.
However, in inference I have to get a new token at a time ...
0
votes
0
answers
5
views
Conversational model returns empty string after a while
I've been experinmenting with Huggingface models and I've set up a chatbot with DialoGPT. It works pretty well, but after a while it stops answering and just returns empty strings.
Before this it will ...
1
vote
1
answer
25
views
Comparison between applications of vanilla transformer and BERT
I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are:
what are the applications of ...
2
votes
0
answers
18
views
1D Sequence Classification
Cross-post from https://stackoverflow.com/questions/71752744/1d-sequence-classification
I am working with a long sequence (~60 000 timesteps) classification task with continuous input domain. The ...
0
votes
0
answers
31
views
Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?
I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
0
votes
0
answers
21
views
Special tokens for encoder and decoder in the transformer architecture
I am trying to wrap my head around the different special tokens that the different transformer architectures use.
For example, let's say we have the following input and target both for a text ...
0
votes
0
answers
23
views
Comparison of zero-shot learning, one-shot learning, and few-shot learning?
What are the differences between zero-shot , one-shot , few-shot learning? and what about their difference in usage/ application? fields of their application?
Comparisons of their Pros & Cons?
0
votes
1
answer
54
views
Incorporating structural information in a Transformer?
For a Neural Machine Translation (NMT) task, my input data has relational information. This relation could be modelled using a graphical structure.
So one approach could be to use Graph Neural Network ...
2
votes
1
answer
81
views
Proof that multihead works better than single head in transformer
According to this post, the purpose of the multihead is to have 'gradient splitting' across heads, which is achieved by random initialization of weight matrices for Q, K and V in each head. But how ...
0
votes
1
answer
13
views
What makes differences in each head in the multiheaded attention in transformer?
What makes differences in each head in the multiheaded attention in transformer?
As they are fed and trained in the exact same way, except the initialization of weights are different for each head to ...
0
votes
0
answers
17
views
The separate of K and V is redundant in transformer?
imho, I think the separate of K and V is redundant in transformer, as they are basically the same regardless in encoder self-attention, or decoder self-attention, or even the encoder-decoder attention....
2
votes
0
answers
10
views
Add features to linear layer of transformer
I'm currently using a transformer for a binary text classification task. Besides the raw text I would also like to add some other handpicked features like text length, text date, text author etc.
I ...
0
votes
0
answers
98
views
ValueError: Exception encountered when calling layer "transformer" (type Transformer)
So I code a Transformers neural network that works as an ASR, it works, it trains good and saved the model as...
model.save("savedmodel.model")
The ...
0
votes
1
answer
23
views
No feedback in Transformers
Newbie question about transformers.
I am referring to the paper https://arxiv.org/pdf/1706.03762.pdf .
Figure 1 (bottom-right) says: "Outputs (shifted right)".
To me, during generation (not ...
0
votes
1
answer
20
views
Using KerasClassifier for training neural network
I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
0
votes
1
answer
31
views
What happens when the length of input is shorter than length of output in transformer architecture?
Given standard transformer architecture with encoder and decoder.
What happens when the input for the encoder is shorter than the expected output from the decoder?
The decoder is expecting to receive ...
0
votes
0
answers
21
views
What is the effect and How "MATMUL" works for Query and Key in Self Attention in Transformer Encoder architecture?
I am learning about the Transformer Architecture: Attention is all you need by coding it from scratch in pytorch. I got this awesome video on Youtube explaning how ...
0
votes
1
answer
18
views
Transformer similarity fine-tuned way too often predicts pairs as similar
I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data:
...
0
votes
0
answers
20
views
Finetune XLM-RoBERTa on Tensorflow
I want to finetune pre-trained XLM-RoBERTa from HuggingFace for Text classification. I have categorical data in English. I want to finetune model on Tensorflow-keras. Can anyone let me know how can I ...
1
vote
1
answer
64
views
How to prepare texts to BERT/RoBERTa models?
I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences.
I want to train a language ...
0
votes
0
answers
118
views
Finetune XLM-RoBERTa on TF-keras for text classification
I am trying to finetune pre-trained XLM-RoBERTa on Tensorflow-keras. I am using dataset in English for text classification. I have used xlm-roberta-base tokenizer to tokenize the sentences. I am using ...
0
votes
0
answers
51
views
How Transformer Decoders Use Mask to Prevent Ground Truth Leakage During Training process
After reading the original paper and many articles and blogs, I have a general understanding of Transformer.
I still have some doubts about the Mask, I know it is to prevent the subsequent positions ...
0
votes
1
answer
96
views
Positional encoding without input embedding
Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer?
In NLP models, the embedding maps a word to real numbers. ...
0
votes
0
answers
20
views
Deep Learning: which kind of layer for {time_series + static} -> time_series problems?
as you can see in the image below, I need to bridge static data (on the left) and time series data (on the right) to create a time series output. I have looked at this example on keras library which ...
0
votes
0
answers
42
views
A minimal example of text generation with nn.Transformer
I am learning PyTorch. My goal is to find a code/tutorial/lesson that creates a text generator (not an lstm or text translator) nn.Transformer. After looking at colab tutorial, I saw a detailed ...
0
votes
0
answers
16
views
How was the vocab built using WordPiece for the paper exBERT?
In the paper exBERT: Extending Pre-trained Models with Domain-specific Vocabulary
Under Constrained Training Resources the authors point out that:
First, we derive an extension vocabulary from the
...
0
votes
1
answer
34
views
Sequence-to-Sequence Transformer for Neural machine translation
I am using the tutorial in Keras documentation here. I am new to deep learning. On a different dataset Menyo-20k dataset, of about 10071 total pairs, 7051 training ...
0
votes
0
answers
46
views
Fine tune t5-small for text summarization
I am trying to fine tune t5-small for text summarization, and I have the following graph loss per batch:
and learning rate per batch:
Do you think loss graph is normal (regarding this use case) or ...
0
votes
0
answers
235
views
HuggingFace transformer: CUDA out memory only when performing hyperparameter search
I am working with a GTX3070, which only has 8GB of GPU RAM. When I am running using trainer.train(), I run fine with a maximum batch size of 7 (6 if running in Jupiter notebook). However, when I ...
0
votes
1
answer
23
views
Get sentence embeddings of transformer-based models
I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).
I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
2
votes
0
answers
13
views
Best practices to train a transformer text classifier to predict/handle unseen labels
I fine-tuned a RoBERTa sequence classifier to classify paragraphs of certain documents using labeled paragraphs only (and skipping paragraphs with no label given). The model was validated and tested ...
0
votes
1
answer
349
views
Difference between Doc2Vec and BERT
I am trying to understand the difference between Doc2Vec and BERT. I do understand that doc2vec uses a paragraph ID which also serves as a paragraph vector. I am not sure though if that paragraph ID ...
1
vote
1
answer
813
views
Variational Autoencoders VS Transformers
I'm relatively new to the field, but I'd like to know how do variational autoencoders fare compared to transformers?
0
votes
0
answers
15
views
Algorithms for classification of very short text
I am to create a classification model for texts that typically have 3 to 4 words in them. I thought of using BERT and XLNet but not sure if they are the right choice for texts that short.
Are there ...
0
votes
0
answers
15
views
If Bert can handle only 512 inputs. Why you can provide such long contexts in QA Pipeline?
For example, I use Pipeline from Huggingface Transformers to use a QA model card like this.
...
1
vote
0
answers
46
views
Variable batch size for inputs of different length
We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...