Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
10 views

How to optimize hyperparameters in Bert?

I am using the BERT model in order to classify stereotypes in sentences. I wanted to know if there is a way to automate the optimization of hyperparameters such as 'epochs', 'batchs' or 'learning rate'...
user avatar
  • 320
0 votes
0 answers
30 views

Can i use Transformer-XL for text classification task?

I want to use transformer xl for text classification tasks. But I don't know the architect model for the text classification task. I use dense layers with activation softmax for logits output from the ...
user avatar
  • 1
1 vote
1 answer
22 views

BERT base uncased required gpu ram

I'm working on an NLP task, using BERT, and I have a little doubt about GPU memory. I already made a model (using DistilBERT) since I had out-of-memory problems with tensorflow on a RTX3090 (24gb gpu'...
user avatar
  • 68
0 votes
0 answers
7 views

BertTokenizer on custom data returns same index for all tokens

I'm trying to train Bert tokenizer on a custom dataset but when running tokenizer.tokenize on sample data, it returns the same index for every tokens which is ...
user avatar
  • 247
0 votes
1 answer
11 views

In what ways are transformers better for Vision than CNNs?

I am looking at getting an intuition to how transformers can be better in vision tasks.
user avatar
0 votes
0 answers
12 views

Model stalls and learning slows down after the first epochs

I have an issue with a model that I'm working on, I cannot show you the model architecture because it's basically confidential research. The model includes Graph convolutional networks and Transformer ...
user avatar
0 votes
0 answers
34 views

Transformer time series classification using time2vec positional embedding

I want to use a transformer model to do classification of fixed-length time series. I was following along this tutorial using keras which uses time2vec as a positional embedding. According to the ...
user avatar
0 votes
0 answers
12 views

Creating a new prediction for the Keras.io BST model

The Keras.io example of a Transformer-based recommendation system is a great example for me to understanding neural networks in Keras. But how would you use the create_model_inputs() to get a new ...
user avatar
0 votes
1 answer
16 views

How do i generate text from ids in Torchtext's sentencepiece_numericalizer?

The torchtext sentencepiece_numericalizer() outputs a generator with indices SentencePiece model corresponding to token in the input sentence. From the generator, I ...
user avatar
  • 17
0 votes
0 answers
18 views

Can I use MLM method to fine tune my BERT model, if it was initially trained with natural language inference method?

I am using BERT model for sentence similarity task. However my dataset with sentence is very specific and I want to fine tune my model on it first. My dataset is unlabelled. And BERT model that I want ...
user avatar
  • 113
0 votes
0 answers
23 views

Should I pretrain my BERT model on specific dataset if it has only one class of labels?

I want to use BERT model for sentences similarity measuring task. I know that BERT models were trained with natural language inference architecture with dataset with labels neutral, entailment, ...
user avatar
  • 113
0 votes
0 answers
15 views

While training BERT variant, getting IndexError: index out of range in self

While training XLMRobertaForSequenceClassification: ...
user avatar
  • 137
1 vote
0 answers
15 views

Guide to Natural language Prompt programming for few-shot learning of Pretrained Language Models

I'm currently working on a project with the goal of producing AI content in the space of a content generation like blog writing, Instagram caption generation etc. Found the in-context few-shot ...
user avatar
0 votes
1 answer
25 views

Why can't positions in transformers be simply appended to the input to preserve the positional information instead of using positional encodings?

I saw in an intro to transformers in this video that positional encodings need to be used to preserve positional information, otherwise word order may not be understood by the neural network. They ...
user avatar
0 votes
0 answers
20 views

While doing inference with a Transformer-Decoder in batches, how can I stop each sequence separately (if possible)?

So my decoder is a transformer-decoder and in training I don't have any issue. I have all the input from the beggining and correctly masked. However, in inference I have to get a new token at a time ...
user avatar
0 votes
0 answers
5 views

Conversational model returns empty string after a while

I've been experinmenting with Huggingface models and I've set up a chatbot with DialoGPT. It works pretty well, but after a while it stops answering and just returns empty strings. Before this it will ...
user avatar
  • 1,248
1 vote
1 answer
25 views

Comparison between applications of vanilla transformer and BERT

I try to identify applications of vanilla transformer in nlp, as well as those in BERT. But I don't seem to find good summaries for either of them. Thus my questions are: what are the applications of ...
user avatar
  • 359
2 votes
0 answers
18 views

1D Sequence Classification

Cross-post from https://stackoverflow.com/questions/71752744/1d-sequence-classification I am working with a long sequence (~60 000 timesteps) classification task with continuous input domain. The ...
user avatar
  • 143
0 votes
0 answers
31 views

Do the multiple heads in Multi head attention actually lead to more parameters or different outputs?

I am trying to understand Transformers. While I understand the concept of the encoder-decoder structure and the idea behind self-attention what I am stuck at is the "multi head part" of the &...
user avatar
0 votes
0 answers
21 views

Special tokens for encoder and decoder in the transformer architecture

I am trying to wrap my head around the different special tokens that the different transformer architectures use. For example, let's say we have the following input and target both for a text ...
user avatar
0 votes
0 answers
23 views

Comparison of zero-shot learning, one-shot learning, and few-shot learning?

What are the differences between zero-shot , one-shot , few-shot learning? and what about their difference in usage/ application? fields of their application? Comparisons of their Pros & Cons?
user avatar
  • 1,516
0 votes
1 answer
54 views

Incorporating structural information in a Transformer?

For a Neural Machine Translation (NMT) task, my input data has relational information. This relation could be modelled using a graphical structure. So one approach could be to use Graph Neural Network ...
user avatar
2 votes
1 answer
81 views

Proof that multihead works better than single head in transformer

According to this post, the purpose of the multihead is to have 'gradient splitting' across heads, which is achieved by random initialization of weight matrices for Q, K and V in each head. But how ...
user avatar
  • 359
0 votes
1 answer
13 views

What makes differences in each head in the multiheaded attention in transformer?

What makes differences in each head in the multiheaded attention in transformer? As they are fed and trained in the exact same way, except the initialization of weights are different for each head to ...
user avatar
  • 359
0 votes
0 answers
17 views

The separate of K and V is redundant in transformer?

imho, I think the separate of K and V is redundant in transformer, as they are basically the same regardless in encoder self-attention, or decoder self-attention, or even the encoder-decoder attention....
user avatar
  • 359
2 votes
0 answers
10 views

Add features to linear layer of transformer

I'm currently using a transformer for a binary text classification task. Besides the raw text I would also like to add some other handpicked features like text length, text date, text author etc. I ...
user avatar
0 votes
0 answers
98 views

ValueError: Exception encountered when calling layer "transformer" (type Transformer)

So I code a Transformers neural network that works as an ASR, it works, it trains good and saved the model as... model.save("savedmodel.model") The ...
user avatar
0 votes
1 answer
23 views

No feedback in Transformers

Newbie question about transformers. I am referring to the paper https://arxiv.org/pdf/1706.03762.pdf . Figure 1 (bottom-right) says: "Outputs (shifted right)". To me, during generation (not ...
user avatar
  • 111
0 votes
1 answer
20 views

Using KerasClassifier for training neural network

I created a simple neural network for binary spam/ham text classification using pretrained BERT transformer. The current pure-keras implementation works fine. I wanted however to plot certain metrics ...
user avatar
  • 247
0 votes
1 answer
31 views

What happens when the length of input is shorter than length of output in transformer architecture?

Given standard transformer architecture with encoder and decoder. What happens when the input for the encoder is shorter than the expected output from the decoder? The decoder is expecting to receive ...
user avatar
0 votes
0 answers
21 views

What is the effect and How "MATMUL" works for Query and Key in Self Attention in Transformer Encoder architecture?

I am learning about the Transformer Architecture: Attention is all you need by coding it from scratch in pytorch. I got this awesome video on Youtube explaning how ...
user avatar
  • 313
0 votes
1 answer
18 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
user avatar
  • 101
0 votes
0 answers
20 views

Finetune XLM-RoBERTa on Tensorflow

I want to finetune pre-trained XLM-RoBERTa from HuggingFace for Text classification. I have categorical data in English. I want to finetune model on Tensorflow-keras. Can anyone let me know how can I ...
user avatar
1 vote
1 answer
64 views

How to prepare texts to BERT/RoBERTa models?

I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences. I want to train a language ...
user avatar
  • 141
0 votes
0 answers
118 views

Finetune XLM-RoBERTa on TF-keras for text classification

I am trying to finetune pre-trained XLM-RoBERTa on Tensorflow-keras. I am using dataset in English for text classification. I have used xlm-roberta-base tokenizer to tokenize the sentences. I am using ...
user avatar
0 votes
0 answers
51 views

How Transformer Decoders Use Mask to Prevent Ground Truth Leakage During Training process

After reading the original paper and many articles and blogs, I have a general understanding of Transformer. I still have some doubts about the Mask, I know it is to prevent the subsequent positions ...
user avatar
  • 1
0 votes
1 answer
96 views

Positional encoding without input embedding

Does it make sense to use a positional encoding in attention when the input tokens do not go through an embedding layer? In NLP models, the embedding maps a word to real numbers. ...
user avatar
0 votes
0 answers
20 views

Deep Learning: which kind of layer for {time_series + static} -> time_series problems?

as you can see in the image below, I need to bridge static data (on the left) and time series data (on the right) to create a time series output. I have looked at this example on keras library which ...
user avatar
  • 159
0 votes
0 answers
42 views

A minimal example of text generation with nn.Transformer

I am learning PyTorch. My goal is to find a code/tutorial/lesson that creates a text generator (not an lstm or text translator) nn.Transformer. After looking at colab tutorial, I saw a detailed ...
user avatar
0 votes
0 answers
16 views

How was the vocab built using WordPiece for the paper exBERT?

In the paper exBERT: Extending Pre-trained Models with Domain-specific Vocabulary Under Constrained Training Resources the authors point out that: First, we derive an extension vocabulary from the ...
user avatar
  • 313
0 votes
1 answer
34 views

Sequence-to-Sequence Transformer for Neural machine translation

I am using the tutorial in Keras documentation here. I am new to deep learning. On a different dataset Menyo-20k dataset, of about 10071 total pairs, 7051 training ...
user avatar
  • 17
0 votes
0 answers
46 views

Fine tune t5-small for text summarization

I am trying to fine tune t5-small for text summarization, and I have the following graph loss per batch: and learning rate per batch: Do you think loss graph is normal (regarding this use case) or ...
user avatar
0 votes
0 answers
235 views

HuggingFace transformer: CUDA out memory only when performing hyperparameter search

I am working with a GTX3070, which only has 8GB of GPU RAM. When I am running using trainer.train(), I run fine with a maximum batch size of 7 (6 if running in Jupiter notebook). However, when I ...
user avatar
0 votes
1 answer
23 views

Get sentence embeddings of transformer-based models

I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...). I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
user avatar
  • 145
2 votes
0 answers
13 views

Best practices to train a transformer text classifier to predict/handle unseen labels

I fine-tuned a RoBERTa sequence classifier to classify paragraphs of certain documents using labeled paragraphs only (and skipping paragraphs with no label given). The model was validated and tested ...
user avatar
  • 21
0 votes
1 answer
349 views

Difference between Doc2Vec and BERT

I am trying to understand the difference between Doc2Vec and BERT. I do understand that doc2vec uses a paragraph ID which also serves as a paragraph vector. I am not sure though if that paragraph ID ...
user avatar
  • 23
1 vote
1 answer
813 views

Variational Autoencoders VS Transformers

I'm relatively new to the field, but I'd like to know how do variational autoencoders fare compared to transformers?
user avatar
0 votes
0 answers
15 views

Algorithms for classification of very short text

I am to create a classification model for texts that typically have 3 to 4 words in them. I thought of using BERT and XLNet but not sure if they are the right choice for texts that short. Are there ...
user avatar
0 votes
0 answers
15 views

If Bert can handle only 512 inputs. Why you can provide such long contexts in QA Pipeline?

For example, I use Pipeline from Huggingface Transformers to use a QA model card like this. ...
user avatar
1 vote
0 answers
46 views

Variable batch size for inputs of different length

We're fine-tuning a GPT-2 model (using the Adam optimizer) to some posts from a social network. The length of these posts varies quite dramatically, so while some are only two tokens long, others can ...
user avatar

1
2 3 4 5 6