Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
330
questions
1
vote
1
answer
12
views
Should I annotate additional information besides the categories I already need in a text?
I have a dataset with bank transfer reasons. They vary a lot because humans wrote them.
From the reasons that are linked to invoice payments I need to extract several things:
invoice number(s)
IBAN
...
- 153
0
votes
0
answers
9
views
State of the art in transformer regressors/attention layer aggregation
I am curious to know what the state of the art is in using transformers for regression. Ultimately what I am interested in is how researchers in this field aggregate the outputs of the final attention ...
- 21
0
votes
1
answer
36
views
How do transformers differ from feature selection and regular machine learning?
This is perhaps a simplistic way of thinking, but to me transformers (attention based neural networks) focus on a subset of the input, learning what is important for the problem/prediction as the ...
- 71
0
votes
1
answer
48
views
One word changes everything NLP
I have a classification model (BERT) that classifies sentences as either question or normal sentences. But whenever a sentence has "how" word, the model chooses "question" class.
...
- 87
0
votes
1
answer
16
views
Inference Process in Autoregressive Transformer Architecture
I'm abit confused about how the inference/prediction process works in transformer.
For simplicity suppose it is a transformer for translation. My understanding is that when training, the whole input ...
1
vote
1
answer
37
views
Threshold determination / prediction for cosine similarity scores
Given a query sentence, we search and find similar sentences in our corpus using transformer-based models for semantic textual similarity.
For one query sentence, we might get 200 similar sentences ...
- 111
2
votes
1
answer
22
views
Ignore or predict padding
I have a sequence to sequence classification model with two classes (similar to NER transformer) and because my data samples have different lengths I use padding. Is it better to use a custom loss ...
- 47
1
vote
1
answer
15
views
What could cause pre-trained Opus-MT models have wildly varying inference time when being used with transformers library?
I have been testing pre-trained Opus-MT models ported to transformers library for python implementation. Specifically, I am using opus-mt-en-fr for English to French translation. And the tokenizer and ...
- 131
0
votes
1
answer
28
views
Multi head self attention output size for batches with different sequence length
I have a question regarding the self attention layer of transformers. When dealing with sequences of varying lengths in a mini-batch, we pad sequences so that all sequences in the batch have the same ...
- 3
0
votes
0
answers
8
views
When to use "contextual" embeddings from language models?
When we use Word2Vec, it's obviously a non-contextual embedding because every word has the same representation. When I pass it to my LSTM, we say the hidden states are the contextual embeddings of the ...
- 109
0
votes
0
answers
12
views
Is attention cache useful during transformer pretraining?
I am looking at the MegatronLM implementation, and the only thing that is cached are the results of xK and xV computation:
https:...
- 111
0
votes
0
answers
11
views
Binary classification of video segment
Are there models that take a small video stream of 5-10 seconds as input (not a single frame, specifically video), and predict is there was an some kind of anomaly in this video segment or not?
2
votes
1
answer
26
views
How to further fine-tune a transformer NLP model on domain specific dataset, after general fine-tuning
I would like to fine-tune a pre-trained BERT-like model for a semantic similarity analysis task in the fashion of the SNLI/MNLI task (i.e. classify sentence pairs to "entailment" or "...
- 41
0
votes
0
answers
26
views
Dataset Format for fine tuning deepset/roberta-base-squad2 hugging face transformer model
I have been trying to fine tune the roberta model for QnA to my specific domain (healthcare).
I am unable to find the correct way to provide the dataset format to the tokenizer in order to fine tune ...
0
votes
0
answers
20
views
Basic doubt on embeddings by BERT, LSTM
When we use Word2Vec, Its obviously a non contextual embedding because every word has a same representation. When I pass it to my LSTM, We say the hidden states are the contextual embeddings of the ...
- 109
1
vote
2
answers
48
views
Extend BERT or any transformer model using manual features
I have been doing a thesis in my citation classifications. I just implemented Bert model for the classification of citations. I have 4 output classes and I give an input sentence and my model returns ...
- 11
0
votes
1
answer
20
views
Text cleaning when applying Sentence Similarity / Semantic Search
Do we need to apply text cleaning practices for the task of sentence similarity?
Most models are being used with whole sentences that even have punctuation. Here are two example sentences that we wish ...
- 111
0
votes
0
answers
15
views
Surrogate model for [parameter vector] to [time series]
Say I have a model $M$ that takes in a parameter vector $\beta$, and produces a (numerical) time series. This could be a complicated model (e.g. a bespoke enzyme reaction model), or something simple ...
- 13
0
votes
0
answers
11
views
What does logits in Casual Language Modeling represent?
I am reading the docs for transformers by hugging face and I see that the logits produced by casual language models are of the shape (batch_size, sequence_length, config.vocab_size).
I also read the ...
0
votes
0
answers
8
views
What are the architectures that CAN use Vision Transformers as a backbone for Object Detection?
I just want to know that how can I fine tune Vision Transformer like DiT for object detection on my custom data and classes?
I want to train a model for Document ...
- 313
0
votes
0
answers
6
views
How to use fairness methods in Bert text classification problem
I have a dataset from HatexPlain for hate speech research, which includes tweets text data with some 'potential targets' that the text is targeting. I want to investigate the fairness property in the ...
- 13
0
votes
0
answers
12
views
How to use a `lr_scheduler` when you don't known how many training steps to do?
I am trying to fine-tune a BERT model, but instead of doing it a fix number of training steps an stalling policy and allow it to run until the model stalls for N evaluations. However, I was previously ...
- 101
0
votes
0
answers
9
views
Intuitive explanation for summing the embedding and positional encoding in the Transformer's embedding
In the Transformer model, the embedding and positional encoding are summed together to represent a word in each location ('positional embedding' from now on).
This way, each cell contains semantic and ...
- 179
0
votes
0
answers
20
views
Is huggingface classification architecture essentially a sequential neural network once tokens are vectorized?
I am a researcher in long-term care in the UK. I have trained a text classification model to identify loneliness in older people using the awesome transformers ...
- 183
0
votes
0
answers
16
views
Bare minimum Transformer (Keras)
I'm experimenting with Transformers for a time-series prediction problem, where inputs are $k$-length sequences and outputs are scalar values.
Looking at the Keras Transformer example, my question is -...
- 101
0
votes
1
answer
25
views
Running Model on both GPUs and CPUs
I have access to a hpc node, of 3 GPU and maximum of 38 CPU. I have a transformer model which I run of a single GPU at the moment, I want to utilize all the GPUs and CPUs.
I have seen couple of ...
- 17
2
votes
0
answers
17
views
how Can we add extra word embedding to the pytorch funnel transformer?
i was approaching NLP sequence classification problem (3 classes) using huggingface transformers (funnel-transformer/large) and tensorflow.
first i created laserembedding like this :
...
- 121
0
votes
0
answers
36
views
masking time series data for unsupervised learning
I was studing the A TRANSFORMER-BASED FRAMEWORK FOR MULTIVARIATE TIME SERIES REPRESENTATION LEARNING
they frame the unsupervised pre-training as the autoregressive task of denoising the multivariate ...
- 101
0
votes
2
answers
72
views
Creating class labels for custom DataSets efficiently (HuggingFace)
I have pandas dataframes - test & train,they both have text and label as columns as shown below -
...
- 103
2
votes
1
answer
31
views
What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?
I am building a project for my bachelor thesis and am wondering how to prepare my raw data. The goal is to program some kind of semantic search for job postings. My data set consists of stored web ...
- 23
0
votes
1
answer
20
views
Do I need training data in multiple languages for a multilingual transformer?
I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case.
...
- 189
0
votes
0
answers
9
views
Using a supervised model (BiGRU-CRF) with lack of labelled data. How to get labels with no human labeled data?
I'm working with a project supervisor on a deep learning project. The project involves extracting keywords or catchphrases from legal documents so that they can then be used for semantic search of ...
0
votes
1
answer
36
views
Transformers vs RNN basic doubt
I have a basic doubt. Kindly clarify this.
My doubt is, When we are using LSTM's, We pass the words sequentially and get some hidden representations.
Now transformers also does the same thing except ...
- 109
0
votes
0
answers
27
views
Which model is best for the generating accurate answers of the Boolean questions?
I am trying to generate the question using T5 transformer answer of the questions but I am getting the error like below.
here is the code.
...
0
votes
1
answer
60
views
Smaller embedding size causes lower loss
When I convert my multilingual transformer model to a single lingual transformer model (got my languages embedding from the multilingual transformer and deleted other embeddings, decreased dimensions ...
- 87
0
votes
1
answer
50
views
What to do with Transformer Encoder output?
I'm in the middle of learning about Transformer layers, and I feel like I've got enough of the general idea behind them to be dangerous. I'm designing a neural network and my team would like to ...
- 3
0
votes
0
answers
9
views
Tracing the source: Which reference did the idea of Transformer's Key-query-value come from？
Since Transformers was proposed in 2017, there have been various interpretation schemes about KQV, but the original text does not seem to explain in detail what this KQV is inspired by. I don't need ...
- 101
0
votes
1
answer
24
views
Transformers - Why Self Attention calculate dot product of q and k from of same word?
As far as I understand and looked into Attention Is All You Need and Transformer model for language understanding, the Self Attention at Scaled Dot-Product Attention is calculating $query$ and $key$ ...
- 513
0
votes
1
answer
21
views
SkLearn DecisionTree doesn't include numerical variables after one hot encoding pipeline
I'm trying to fit a dataframe with SkLearn DecisionTree with the following code. But I get a error Length of feature_names, 9 does not match number of features, 8. ...
0
votes
1
answer
127
views
How to get all 3 labels' sentiment from finbert instead of the most likely label's?
I'm using bert to do sentiment analysis. I previous used cardiffnlp's twitter-roberta-base-sentiment, https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment.
It gives the the usage on its ...
- 163
0
votes
1
answer
48
views
What did Sentence-Bert return here?
I used sentence bert to embed sentences from this tutorial https://www.sbert.net/docs/pretrained_models.html
...
- 163
0
votes
0
answers
3
views
Is there practice to train language-to-code transformer (multi-modal transformer) using uni-modal pretrained models-transformers?
Language-to-code transformation/generation require multiple skills - language and reasoning skills to digest the core problem from the natural language specification. And programming language ...
- 131
0
votes
1
answer
17
views
Visualize attention area
I wonder how people draw a network's attention area on a single input.
Such as:
Any hint is much appreciated
- 13
1
vote
1
answer
20
views
How does compute required scale with number of model parameters?
GPT-3 has 175 billion parameters, required ~$3.114 * 10^{23}$ FLOPS, and took approximately one month to train on ~10k Tesla V100 GPUs. It seems commonly stated that the brain has the equivalent of ~...
0
votes
0
answers
32
views
How to get attention weights from BERT
I am trying to make transfer learning in the encoder and decoder parts of the transformer. The encoder is a whole feature extraction part and there is the feature extraction part in the decoder too(I ...
- 87
0
votes
0
answers
64
views
Using BERT embeddings as input for transformer architecture
I will use BERT's embedding weights (as discussed here) for embedding in embedding layers of the transformer model. But my question is: don't embeddings of BERT already go through the whole encoding ...
- 87
0
votes
0
answers
20
views
Transfer Learning transformer architecture
I would like to make transfer learning for transformer architecture. The input of the encoder and decoder must be word embeddings. So I wanted to use a pre-trained BERT model as a word embedder and ...
- 87
0
votes
0
answers
56
views
What are the inputs of encoder and decoder layers of transformer architecture?
In the paper (attention is all you need), it says "embeddings" are the input of the encoding layer. As I know embeddings are the numerical representation of words which is (for example) the ...
- 87
0
votes
0
answers
20
views
Mix of time-dependent and constant features for a transformer
I'm using the transformer architecture to predict future time-points from previous time-points.
Each item of the input sequence is a vector of ...
0
votes
0
answers
19
views
What is the minimal number of examples for BERT-like language model for the model to train a word
I have heard rumors of a particular count of positive examples that allowed the model to train a given word (or context of it - when talking about MLM) to be ~40. I am wondering though about the ...
- 207