Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
0 votes
0 answers
7 views

How many parameters does the vanilla Transformer have?

The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
  • 161
0 votes
0 answers
28 views

ChatGPT: How to use long texts in prompt?

I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT. I would like to program something ...
0 votes
0 answers
5 views

ner transformers implementation in keras

I am following this keras link https://keras.io/examples/nlp/ner_transformers/ to train my own NER model. I am not clear why we have tags +=1 in the following ...
1 vote
1 answer
24 views

Using BERT to extract a list of words and phrases from documents

I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data. For example my list of words and ...
  • 385
1 vote
1 answer
10 views

Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?

I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see. We have the following kind of ...
0 votes
0 answers
12 views

What models are able to handle variable input lengths?

I am tasked with creating a classifier that is able to predict whether an item will be returned. This is supposed to not only happen on the basis of an individual item, but on all other items within ...
0 votes
0 answers
10 views

What is the most efficient way of image document classification?

So, I am working on a project where I have to extract sales tax invoice from the pdf document which contains other files along with the invoice. I researched on the topic, and am considering two ...
1 vote
0 answers
15 views

A French version of Rebel

Is there an end-to-end trained transformer like Rebel for french data? Rebel can extract entities and relations from text, yet as far as I know, it works only with english texts. Is there any other ...
  • 11
2 votes
2 answers
25 views

Should I open abbreviations/acronyms in the text data, when training transformer model?

I am currently training a transformer model on text data. Is it a good practise to open abbreviations/acronyms in the text data? I did not dins any tips or recommendations about it on internet.
  • 183
1 vote
0 answers
9 views

NANs, Infinities, and very large losses with normalizing flows

I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
  • 31
1 vote
2 answers
41 views

What does the output of an encoder in encoder-decoder model represent?

So in most blogs or books touching upon the topic of encoder-decoder architectures the authors usually say that the last hidden state(s) of the encoder is passed as input to the decoder and the ...
  • 113
1 vote
1 answer
41 views

Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?

By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)  Still, I got a question from reading there, hopefully, you guys can help me out  Are the ...
  • 113
0 votes
1 answer
47 views

What are the advantages of autoregressive over seq2seq?

Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture? I know the difference between the attention of ...
0 votes
0 answers
21 views

Which algorithms are suitable for time series classification for this form of data?

Newbie here, I have a large list of csv files that contain a series of probability distributions (for 5 classes). I'm trying to train a binary classifier that classifies each file into either a ...
0 votes
0 answers
25 views

is the distilRoberta transformer model overfitting or underfitting?

I am a bit new to ML, below are the results after I fine tuned distilRoBERTa using HuggingFace Trainer. I cant tell if my model is over-fitting, under-fitting or ok? I ran 7 epochs. I think its ...
  • 1
0 votes
0 answers
14 views

Why can a 3x3 depth-wise convolution replace positional encoding in SegFormer?

Context In the SegFormer Paper, the authors use 3x3 depth-wise convolutions instead of positional encodings with the following reasoning: "We instead introduce Mix-FFN where we consider the ...
  • 46
0 votes
1 answer
31 views

What loss function to use for predicting discrete output sequence given a discrete input sequence?

I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say ...
0 votes
0 answers
12 views

Why don't we use binary vectors in positional coding?

I found this article on positional encoding (https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3). But I don't understand when the author says that you have to measure a ...
  • 1
0 votes
1 answer
30 views

Train question answering model with custom dataset

How can I train a question-answering ML model with a custom dataset? I have gathered nearly 110GB of text data, containing documentation manuals for software products and I am looking into different ...
  • 113
1 vote
0 answers
19 views

How much data and computation power do I need to train a machine translation model using Transformer architecture?

I am working right now on creating a dataset to use in creating a machine translation model to translate between two dialects. I have two questions that I am trying to find an answer for: How much ...
0 votes
2 answers
33 views

how can I translate Whisper encodings to SBERT embeddings?

I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by ...
  • 103
0 votes
0 answers
18 views

Transformer doesn't generalize on a time-series data

Data 120 patients. Each patient has 5.7556e+06 samples on average, each sample consists of 5 features stored as a continuous high-frequency (1000Hz) time series. Labels are 13 discrete classes ...
0 votes
1 answer
72 views

Vision Transformer ViT Parameter count

The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021) includes the following table: Can someone explain how they get the parameter counts or where my calculation is ...
  • 111
0 votes
1 answer
17 views

Predicting a next word from a sentence of a different lenght than seen in training

I am building a custom Decoder-only transformer model, which is being trained on the task of Next Word Prediction. The training procedure is analogous to that of chat GPT models - the input to the ...
0 votes
0 answers
10 views

Training a model that maps embedding from (image, text) to text

I have created embedding say A which is created my concatenating embedding of image and embedding of text, that is concat(img_embedding,text_embedding). Now, I have pairs such as (img_embedding,...
  • 13
0 votes
1 answer
91 views

How to extract embeddings from an audio file using wav2vec along with context

I am trying to use wav2vec embeddings from the XLSR model for emotion recognition on the EMODB dataset. How can I extract embeddings using wav2vec? I want to use the XLSR model pre-trained with ...
0 votes
1 answer
49 views

Requirements for variable length output in transformer

I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
1 vote
1 answer
23 views

What is the best approach to deploy N number of ML models as a scalable service in the Cloud?

I've N (~50) number of sentiment models of different languages, which were fine tuned on HggingFace's transformer models. Each of the models as 2-3 GB in size approx. Now, how can I deploy all these ...
0 votes
0 answers
11 views

Predicting same tokens as base BERT model for token classification on custom dataset

I have a custom dataset with custom tag for each token in the text. I want to train a BERT model for classifying each token into its corresponding category. To do ...
0 votes
0 answers
22 views

Memory Error when loading a txt file for an ML model

I am trying to run the Python code below: ...
  • 113
0 votes
0 answers
23 views

Transformer model with same input fed into the encoder and decoder

I need a model to process an list (Tx = T but variable across samples) of vectors to get another list of vecotrs (Ty = Tx = T so also different across different samples). I would like to use the ...
0 votes
1 answer
43 views

What does Embeddings Array Represent in BERT's Feature Extraction?

I am new to academic NLP, and I had been tasked with to use BERT to extract features of a sentence. ...
0 votes
2 answers
40 views

Using transformers positional embedding

Positional embeddings are introduced into a transformer in order to add positional information to a word embedding. Now, suppose we have an existing data embedding that can be for any data domain word/...
  • 231
1 vote
1 answer
42 views

Are there any practical advantages of LSTMs over transformers?

There are a number of articles noting that transformers have significant advantages over "traditional" RNNs like LSTMs. And the industry as a whole have been moving away from LSTMs. My ...
0 votes
0 answers
14 views

Why do transformers operate on discrete sequences?

Why do we need to discretize our input $x$ vectors in transformers? For example, we often employ VQ-VAE's to discretize images to interface with ViTs. Surely, because attention calculation simply ...
  • 197
0 votes
2 answers
27 views

Compound and Complex Sentence Tokenization

I am trying to tokenize sentences of a document for aspect-based sentiment analysis. There are some sentences that consist of more than one topic. For example, " The touch screen is good but the ...
0 votes
1 answer
22 views

Predict the values of variable features over timestamps

HI i am having a dataset which contain timestamps and number of users at that timestamp. Each user has resource values which change per timestamp. How can i make predictions of number of users ...
0 votes
1 answer
100 views

What does the random seed influence in transformers?

I've been doing some experiments with setting different seeds on transformer training, and wanted to understand why I see so much variance. Am I correct in thinking the random seed only influences two ...
0 votes
0 answers
34 views

Make spacy NER model more robust to handle odd product code entity extraction

I am developing a NER model to extract product codes that are all over the place in terms of format and naming convention (AXEWAL719XA, AX-P20XXT-001, etc.). I started with the basic blank spacy('en') ...
  • 101
1 vote
0 answers
24 views

Is normalization of word embeddings important?

I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
0 votes
0 answers
15 views

Finetune an already finetuned transformer model

I have a use case for a model where the backbone is a transformer(ViT-based) and has been pretrained for Masked Image Modeling. The output of the backbone is pushed into an FPN tensor which then Mask ...
2 votes
0 answers
152 views

Adding punctuation for a long text

I want to add punctuation to a long text (youtube transcript) before using a Transformer pipeline for summarization. I have found this answer here: original answer thus I have tried: ...
  • 21
0 votes
1 answer
86 views

Possible NLP approaches to extract 'goals' from text

I am planning to take up an interesting NLP project. I want to extract 'goal' statements from lengthy reports. For example, the goals can be We would be reducing our carbon footprint by 50% by 2025 or ...
3 votes
1 answer
70 views

Solving video classification problem by taking EVA Large as backbone

I am solving a video classification problem. There are 9 classes in total. At first I took ResNet as a feature extractor, this gave me 0.74 accuracy. Then I changed ResNet to EVA (I also tried Swin), ...
  • 131
3 votes
1 answer
44 views

How is padding masking considered in the Attention Head of a Transformer?

For purely educational purposes, my goal is to implement basic Transformer architecture from scratch. So far I focused on the encoder for classification tasks and assumed that all samples in a batch ...
  • 171
0 votes
1 answer
45 views

Is there bias in matrix multiplications for self attention

When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well. ...
  • 13
0 votes
0 answers
22 views

Combining huggingface's transfformers pipeline with pytorch function

I have a series of natural language phrases from which I'd like to extract the average of the word embeddings to get a pseudo-sentence embedding. I'm using a huggingface pipeline to extract the ...
0 votes
0 answers
10 views

How can i increase accuracy of my fine tuned T-5 text summarizer?

I am working on text summarization, I have fine-tuned of T-5 model with my dataset. I am using a small dataset. I have to perform with this dataset. Now I am facing two problems. 1 - Low Accuracy on ...
0 votes
0 answers
45 views

Can I do prompt-learning on HuggingFace Transformers?

I'm trying to solve a Prompt Learning task, but I'm not finding information on how to do it with HuggingFace Transformers. However, I've been thinking about it, and doing prompt-learning is basically ...
2 votes
2 answers
91 views

NER - What advantage does IO Format have over BIO Format

In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or "O" in case the word ...

1
2 3 4 5
8