Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
379
questions
0
votes
0
answers
7
views
How many parameters does the vanilla Transformer have?
The original Transformer paper (Vaswani et al; 2017 NeurIPS) describes the model architecture and the hyperparameters in quite some detail, but it misses to provide the exact (or even rough) model ...
- 161
0
votes
0
answers
28
views
ChatGPT: How to use long texts in prompt?
I like the website chatpdf.com a lot. You can upload a PDF file and then discuss the textual content of the file with the file "itself". It uses ChatGPT.
I would like to program something ...
- 23
0
votes
0
answers
5
views
ner transformers implementation in keras
I am following this keras link https://keras.io/examples/nlp/ner_transformers/
to train my own NER model. I am not clear why we have tags +=1 in the following ...
1
vote
1
answer
24
views
Using BERT to extract a list of words and phrases from documents
I have a list of words and phrases (~3k items). What are my options to extract them from documents (~3M of job descriptions) with NLP? I do not have labeled data.
For example my list of words and ...
- 385
1
vote
1
answer
10
views
Does high number of output labels affect the performance of BERT and how to handle the class imbalance issue while doing multi text classification?
I am using BERT to do multiclass text classification. The number of output classes I have to predict from is: 116 and there is high degree of class imbalance that I see.
We have the following kind of ...
0
votes
0
answers
12
views
What models are able to handle variable input lengths?
I am tasked with creating a classifier that is able to predict whether an item will be returned.
This is supposed to not only happen on the basis of an individual item, but on all other items within ...
0
votes
0
answers
10
views
What is the most efficient way of image document classification?
So, I am working on a project where I have to extract sales tax invoice from the pdf document which contains other files along with the invoice. I researched on the topic, and am considering two ...
1
vote
0
answers
15
views
A French version of Rebel
Is there an end-to-end trained transformer like Rebel for french data?
Rebel can extract entities and relations from text, yet as far as I know, it works only with english texts.
Is there any other ...
- 11
2
votes
2
answers
25
views
Should I open abbreviations/acronyms in the text data, when training transformer model?
I am currently training a transformer model on text data. Is it a good practise to open abbreviations/acronyms in the text data? I did not dins any tips or recommendations about it on internet.
- 183
1
vote
0
answers
9
views
NANs, Infinities, and very large losses with normalizing flows
I am new to normalizing flows and have been trying to use them with a high-dimensional dataset, and I have been running into very large numbers and errors with sampling that don't occur when I use a ...
- 31
1
vote
2
answers
41
views
What does the output of an encoder in encoder-decoder model represent?
So in most blogs or books touching upon the topic of encoder-decoder architectures the authors usually say that the last hidden state(s) of the encoder is passed as input to the decoder and the ...
- 113
1
vote
1
answer
41
views
Self-attention in Transformers, are the component values of input vector trained or is it the set W_q, W_k, W_v?
By far, I find this tutorial on self-attention the most digestible (https://peterbloem.nl/blog/transformers)
Still, I got a question from reading there, hopefully, you guys can help me out
Are the ...
- 113
0
votes
1
answer
47
views
What are the advantages of autoregressive over seq2seq?
Why are recent dialog agents, such as ChatGPT, BlenderBot3, and Sparrow, based on the decoder architecture instead of the encoder-decoder architecture?
I know the difference between the attention of ...
- 3
0
votes
0
answers
21
views
Which algorithms are suitable for time series classification for this form of data?
Newbie here, I have a large list of csv files that contain a series of probability distributions (for 5 classes). I'm trying to train a binary classifier that classifies each file into either a ...
- 101
0
votes
0
answers
25
views
is the distilRoberta transformer model overfitting or underfitting?
I am a bit new to ML, below are the results after I fine tuned distilRoBERTa using HuggingFace Trainer. I cant tell if my model is over-fitting, under-fitting or ok? I ran 7 epochs.
I think its ...
- 1
0
votes
0
answers
14
views
Why can a 3x3 depth-wise convolution replace positional encoding in SegFormer?
Context
In the SegFormer Paper, the authors use 3x3 depth-wise convolutions instead of positional encodings with the following reasoning: "We instead
introduce Mix-FFN where we consider the ...
- 46
0
votes
1
answer
31
views
What loss function to use for predicting discrete output sequence given a discrete input sequence?
I am working on sequence-to-sequence tasks where the input is an n-length sequence of discrete values from a finite set S (say ...
0
votes
0
answers
12
views
Why don't we use binary vectors in positional coding?
I found this article on positional encoding (https://towardsdatascience.com/master-positional-encoding-part-i-63c05d90a0c3).
But I don't understand when the author says that you have to measure a ...
- 1
0
votes
1
answer
30
views
Train question answering model with custom dataset
How can I train a question-answering ML model with a custom dataset?
I have gathered nearly 110GB of text data, containing documentation manuals for software products and I am looking into different ...
- 113
1
vote
0
answers
19
views
How much data and computation power do I need to train a machine translation model using Transformer architecture?
I am working right now on creating a dataset to use in creating a machine translation model to translate between two dialects. I have two questions that I am trying to find an answer for:
How much ...
- 11
0
votes
2
answers
33
views
how can I translate Whisper encodings to SBERT embeddings?
I'm using the Whisper model to recognize speech, and then matching the output text against a list of known questions by generating SBERT embeddings from the text and ranking the known questions by ...
- 103
0
votes
0
answers
18
views
Transformer doesn't generalize on a time-series data
Data
120 patients.
Each patient has 5.7556e+06 samples on average, each sample consists of 5 features stored as a continuous high-frequency (1000Hz) time series.
Labels are 13 discrete classes ...
- 13
0
votes
1
answer
72
views
Vision Transformer ViT Parameter count
The Vision Transformer paper An Image is with 16x16 words by Dosovitskiy et al. (2021)
includes the following table:
Can someone explain how they get the parameter counts or where my calculation is ...
- 111
0
votes
1
answer
17
views
Predicting a next word from a sentence of a different lenght than seen in training
I am building a custom Decoder-only transformer model, which is being trained on the task of Next Word Prediction. The training procedure is analogous to that of chat GPT models - the input to the ...
- 1
0
votes
0
answers
10
views
Training a model that maps embedding from (image, text) to text
I have created embedding say A which is created my concatenating embedding of image and embedding of text, that is concat(img_embedding,text_embedding).
Now, I have pairs such as (img_embedding,...
- 13
0
votes
1
answer
91
views
How to extract embeddings from an audio file using wav2vec along with context
I am trying to use wav2vec embeddings from the XLSR model for emotion recognition on the EMODB dataset. How can I extract embeddings using wav2vec?
I want to use the XLSR model pre-trained with ...
- 23
0
votes
1
answer
49
views
Requirements for variable length output in transformer
I have been working on modifying the transformer from the article The Annotated Transformer. One of the features I would like to include is the ability to pass a sequence of fixed length, and receive ...
- 1
1
vote
1
answer
23
views
What is the best approach to deploy N number of ML models as a scalable service in the Cloud?
I've N (~50) number of sentiment models of different languages, which were fine tuned on HggingFace's transformer models. Each of the models as 2-3 GB in size approx. Now, how can I deploy all these ...
0
votes
0
answers
11
views
Predicting same tokens as base BERT model for token classification on custom dataset
I have a custom dataset with custom tag for each token in the text. I want to train a BERT model for classifying each token into its corresponding category. To do ...
0
votes
0
answers
22
views
Memory Error when loading a txt file for an ML model
I am trying to run the Python code below:
...
- 113
0
votes
0
answers
23
views
Transformer model with same input fed into the encoder and decoder
I need a model to process an list (Tx = T but variable across samples) of vectors to get another list of vecotrs (Ty = Tx = T so also different across different samples). I would like to use the ...
- 101
0
votes
1
answer
43
views
What does Embeddings Array Represent in BERT's Feature Extraction?
I am new to academic NLP, and I had been tasked with to use BERT to extract features of a sentence.
...
- 23
0
votes
2
answers
40
views
Using transformers positional embedding
Positional embeddings are introduced into a transformer in order to add positional information to a word embedding.
Now, suppose we have an existing data embedding that can be for any data domain word/...
- 231
1
vote
1
answer
42
views
Are there any practical advantages of LSTMs over transformers?
There are a number of articles noting that transformers have significant advantages over "traditional" RNNs like LSTMs. And the industry as a whole have been moving away from LSTMs. My ...
- 11
0
votes
0
answers
14
views
Why do transformers operate on discrete sequences?
Why do we need to discretize our input $x$ vectors in transformers? For example, we often employ VQ-VAE's to discretize images to interface with ViTs.
Surely, because attention calculation simply ...
- 197
0
votes
2
answers
27
views
Compound and Complex Sentence Tokenization
I am trying to tokenize sentences of a document for aspect-based sentiment analysis. There are some sentences that consist of more than one topic. For example, " The touch screen is good but the ...
0
votes
1
answer
22
views
Predict the values of variable features over timestamps
HI i am having a dataset which contain timestamps and number of users at that timestamp. Each user has resource values which change per timestamp. How can i make predictions of number of users ...
0
votes
1
answer
100
views
What does the random seed influence in transformers?
I've been doing some experiments with setting different seeds on transformer training, and wanted to understand why I see so much variance.
Am I correct in thinking the random seed only influences two ...
- 772
0
votes
0
answers
34
views
Make spacy NER model more robust to handle odd product code entity extraction
I am developing a NER model to extract product codes that are all over the place in terms of format and naming convention (AXEWAL719XA, AX-P20XXT-001, etc.). I started with the basic blank spacy('en') ...
- 101
1
vote
0
answers
24
views
Is normalization of word embeddings important?
I am doing actor-critic reinforcement learning for an environment that is best represented as a "bag-of-words". For this reason, I have opted to use a single body, multi-head approach for ...
0
votes
0
answers
15
views
Finetune an already finetuned transformer model
I have a use case for a model where the backbone is a transformer(ViT-based) and has been pretrained for Masked Image Modeling. The output of the backbone is pushed into an FPN tensor which then Mask ...
- 878
2
votes
0
answers
152
views
Adding punctuation for a long text
I want to add punctuation to a long text (youtube transcript) before using a Transformer pipeline for summarization.
I have found this answer here:
original answer
thus I have tried:
...
- 21
0
votes
1
answer
86
views
Possible NLP approaches to extract 'goals' from text
I am planning to take up an interesting NLP project. I want to extract 'goal' statements from lengthy reports. For example, the goals can be We would be reducing our carbon footprint by 50% by 2025 or ...
- 101
3
votes
1
answer
70
views
Solving video classification problem by taking EVA Large as backbone
I am solving a video classification problem. There are 9 classes in total. At first I took ResNet as a feature extractor, this gave me 0.74 accuracy. Then I changed ResNet to EVA (I also tried Swin), ...
- 131
3
votes
1
answer
44
views
How is padding masking considered in the Attention Head of a Transformer?
For purely educational purposes, my goal is to implement basic Transformer architecture from scratch. So far I focused on the encoder for classification tasks and assumed that all samples in a batch ...
- 171
0
votes
1
answer
45
views
Is there bias in matrix multiplications for self attention
When the query matrix Q is computed as $XW_Q$, ($W_Q$ is the weight matrix for the queries), is it implemented as a linear layer without bias? I see some blogs saying there is are bias terms as well.
...
- 13
0
votes
0
answers
22
views
Combining huggingface's transfformers pipeline with pytorch function
I have a series of natural language phrases from which I'd like to extract the average of the word embeddings to get a pseudo-sentence embedding. I'm using a huggingface pipeline to extract the ...
0
votes
0
answers
10
views
How can i increase accuracy of my fine tuned T-5 text summarizer?
I am working on text summarization, I have fine-tuned of T-5 model with my dataset. I am using a small dataset. I have to perform with this dataset. Now I am facing two problems.
1 - Low Accuracy on ...
0
votes
0
answers
45
views
Can I do prompt-learning on HuggingFace Transformers?
I'm trying to solve a Prompt Learning task, but I'm not finding information on how to do it with HuggingFace Transformers. However, I've been thinking about it, and doing prompt-learning is basically ...
2
votes
2
answers
91
views
NER - What advantage does IO Format have over BIO Format
In this paper, the authors say that they used IO schema instead of BIO in their dataset, which, if I am not wrong, means they just tag the corresponding Entity Type or "O" in case the word ...
- 23