Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
230
questions
1
vote
1answer
12 views
How does T5 model work on input and target data while transfer learning?
I am working on a project where I want the model to generate job description based on Role, Industry, Skills. I have trained my data and got the resultant output.
I ...
1
vote
1answer
23 views
Combining models trained on a multilingual multi-source corpus
Consider the following training corpora:
dataset1: composed of French instances
dataset2: dataset1 + Arabic instances
test_dataset (for both scenarios): composed of French instances
(the same ...
2
votes
1answer
33 views
Why do Transformers need positional encodings?
At least in the first self-attention layer in the encoder, inputs have a correspondence with outputs, I have the following questions.
Isn't ordering already implicitly captured by the query vectors, ...
0
votes
0answers
10 views
What are the evaluation metrics for evaluating testing data
I am doing a project using T5 Transformer. I have read documentations related to T5 Transformer model. The aim of the project is to generate Job Description based on Job_Role and Skills. My concern ...
0
votes
1answer
12 views
What is the difference between batch_encode_plus() and encode_plus()
I am doing a project using T5 Transformer. I have read documentations related to T5 Transformer model. While using T5Tokenizer I am kind of confused with tokenizing my sentences.
Can someone please ...
0
votes
0answers
4 views
Huggingface - TypeError: 'TensorSliceDataset' object is not subscriptable
I'm trying to make my own model for translate a language to another with T5ForConditionalGeneration and Huggingface using no pretrained model (I need to use my own dataset and tokenizer because no ...
0
votes
0answers
7 views
Custom vocabulary with Transformer model in Trax
I'm currently working on a programming language translation problem (NMT) using a transformer model with Trax. So I basically need to convert a language A to a language B. In order to tokenize the ...
1
vote
0answers
18 views
NER prections with distilbert transformer model
I am trying to extract 'agreement date' label from a corpus of legal contracts. In the train dataset, I used pytorch-transformer model to train.
...
1
vote
0answers
13 views
Can a reformer model really handle long-range dependency?
I read this article about new attention model called Reformer. Here is the main strength of this model:
The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
0
votes
0answers
21 views
How to generate sentences based on words?
I have a dataframe which has columns Role Name, Technical Skills, Soft Skills and average experience. I have to use these words ...
0
votes
1answer
21 views
Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)
BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
0
votes
1answer
10 views
Training NMT models for noisy social media roman text
I am trying to train an NMT model where the source side is roman text of Asian languages from social media, and target side is English. Note that since roman text is not native to Asia, the ...
0
votes
2answers
202 views
AttributeError: 'numpy.ndarray' object has no attribute 'fit' [closed]
I am relatively new to ML and in the process of learning pipelines.
I am creating a pipeline of custom transformers and get this error:
AttributeError: 'numpy.ndarray' object has no attribute 'fit'. ...
0
votes
1answer
8 views
Where do Q vectors come from in Attention-based Sequence-to-Sequence Transformers?
I'm taking a course on Attention-based NLP but I'm not understanding the calculation and application of Attention, based on the use of Q, K, and V vectors. My understanding is that the K and V ...
1
vote
1answer
20 views
Developing a deep learning hybrid architecture for a particular problem is a highly complicated task [closed]
I am currently conducting research on application of deep learning (sensor signal recognition). I spent about a year and a half sifting through the literature and discovered some research patterns. To ...
0
votes
0answers
15 views
How to use is_split_into_words with Huggingface NER pipeline
I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train.
My incoming text has already been split into words. When tokenizing during ...
1
vote
0answers
10 views
What is the principial difference between zero-shot learning and k-NN and clusterization based methods?
One can consider clustering and k-NN to be a zero-shot, too?
I think there is no much principal difference, except using some neural network architecture (usually it is a transformer) which is used to ...
0
votes
0answers
9 views
One single-batch training on Huggingface Bert model "ruins" the model
For some reason, I need to do further (2nd-stage) pre-training on Huggingface Bert model, and I find my training outcome is very bad.
After debugging for hours, surprisingly, I find even training one ...
0
votes
0answers
12 views
Optimal method(s) to monitor attention matrices when doing training/inference using BERT-type models from transformers
Our team is using BERT/Roberta from the huggingface transformers library for sequence-classification (amongst other tasks). We are looking for an efficient way to monitor the attention matrices so as ...
0
votes
2answers
87 views
Please explain Transformer vs LSTM using a sequence prediction example
I don't understand the difference in mechanics of a transformer vs LSTM for a sequence prediction problem. Here is what I have gathered so far:
LSTM:
suppose we want to predict the remaining tokens ...
0
votes
0answers
9 views
How RASA DietClassifier is so fast?
As per RASA DietClassifer architecture, it uses transformer before CRF. Also, it optionally can use SpacyNLP. Despite such a complex model with so many layers the execution speed of RASA is very good.
...
1
vote
1answer
16 views
Transformer model is very slow and doesn't predict well
I created my first transformer model, after having worked so far with LSTMs. I created it for multivariate time series predictions - I have 10 different meteorological features (temperature, humidity, ...
2
votes
0answers
58 views
VQ-GAN understanding
I tried to understand how VQ-GAN works, but unfortunately I have not understood it. I tried to read some articles about it and watch a video. I believe a good and simple article will help me. You ...
0
votes
0answers
13 views
Represent Neural Network as matrix calculation (Transformer Feed Forward NN)
for better understanding, I would like to represent the calculations in a neural network with one hidden layer and one output layer as a matrix calculation.
The hidden layer has 3072 neurons, the ...
0
votes
0answers
13 views
How are the linear fully connected layers trained in transformers
I've been researching into lots of machine learning papers, and I have really been fascinated by the complexity and expressiveness of BERT NLP models.
Therefore, I thought I'd learn the core of ...
0
votes
0answers
30 views
Not clear about relative position bias
I've been reading the Swin Transformer paper and came across relative position bias concept. I'm not able to figure out how is it more effective than positional embeddings. I hope someone can explain ...
0
votes
0answers
17 views
Complexity calculation in Swin Transformer
In Swin Transformer paper, the complexity of MSA and W-MSA is given as:
I have a question regarding 4hwC^2 in both equations. I feel that it should be 3hwC^2 since the computation is for query, key, ...
1
vote
1answer
36 views
How is attention different from linear MLPs?
Each output for both the attention layer (as in transformers) and MLPs or feedforward layer(linear-activation) are weighted sums of previous layer. So how they are different?
1
vote
1answer
23 views
Row embedding as output of a transformer - how are they defined?
I am reading the paper Tabular transformers for modeling multivariate time series and am having issues understanding the structure in Fig. 2. In Sec. 2.2, the authors say that the field transformer ...
1
vote
2answers
16 views
Paraphrasing a sentence and changing the tone of it
I am trying to make a model that is capable of translating a sentence into a new and a better form. I would like the model to change the tone and also give it some character. I am using this in my web ...
0
votes
0answers
27 views
Train a transformer for classification task using time series data
I need to implement and train a Transformer-based model to classify users (binary classification) based on some time-series data. For each user, time-series data are stored as a variable number of ...
0
votes
0answers
7 views
Proper scaling method for time series classification transformer models?
I've asked a similar question about Gradient Boosting Machines already some time ago. This time, I would like to perform time series classifications with a transformer model. I found this Keras ...
0
votes
0answers
30 views
Are there any ways to check default values of pre-trained models before fine-tuning?
Background
According to the instruction on Hugging Face page, I'm trying to fine tune pre-trained model for named entity recognition.
I think I should try Transfer Learning for the first, but there is ...
0
votes
1answer
17 views
An issue for sub-word tokenization preprocessing transformer
I'm stacked with executing the sub-word tokenization preprocessing to use transformer.
According to the tutorial on the article, I have executed the sample code.
However, one function was not defined ...
0
votes
1answer
29 views
Sub-word tokenization preprocessing to use transformer
I'm stacked with executing the sub-word tokenization preprocessing to use transformer.
According to the tutorial on the article, I have executed the sample code.
However, one function was not defined ...
0
votes
0answers
9 views
How to frame queries and answers from customer Agent utterances using Deep Learning SOTA
I am working with smart-reply use case for Async chat customer and ...
0
votes
1answer
46 views
Error to load a pre-trained BERT model
Background
I'm reading this article about a natural language task, named entity recognition and trying to load a pre-trained BERT model on Google colaboratory.
How can I fix an error to load a pre-...
0
votes
0answers
12 views
Can someone explain Minimum Bayes Risk intuituvely? (Explain like I'm five)
I am learning Transformer and studying Decoding, such as beam search and random sampling which are easy to understand. However, when it comes to Minimum Bayes Risk, it is more difficult. Please help
0
votes
0answers
28 views
How to load fine-tuned Electra (TFElectraForSequenceClassification) Model?
I have fine-tuned an Electra Model using the following code.
...
0
votes
1answer
73 views
time series anomaly detection
I want to ask for time series anomaly detection we can apply tnn on multiple features or not?
I used transformer for sentiment analysis where I have to provide a sentence and it predicts its output as ...
0
votes
1answer
46 views
Self-Attention Summation and Loss of Information
In self-attention, the attention for a word is calculated as:
$$
A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>}
$$
My question is why we sum over the ...
0
votes
0answers
53 views
Getting Word Embeddings for Sentences using long-former model?
I am new to Huggingface and have few basic queries. This post might be helpful to others as well who are starting to use longformer model from huggingface.
Objective:
Create Sentence/document ...
0
votes
0answers
34 views
Can we add positional encoding to time series input for time series prediction?
I want to use classical machine learning models such XGBoost for my time series prediction. Since the input data for XGBoost/sklearn based models is 2d i.e. (n_samples, n_features), I want to encode ...
0
votes
1answer
30 views
How do the linear + softmax layers give out word probabilities in Transformer network?
I am trying to implement a transformer network from scratch in pytorch to understand it. I am using The illustrated transformer for guidance. The part where I am stuck is about how do we go from the ...
2
votes
1answer
34 views
BERT Optimization for Production
I'm using BERT to transform text into 768 dim vector, It's multilingual :
...
1
vote
2answers
776 views
What is the difference between BERT and Roberta
I want to understand the difference between BERT and Roberta. I saw the article below.
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
It mentions that ...
0
votes
0answers
10 views
How to prepare data for TpyTorch's 3d attn_mask argument in MultiHeadAttention
I'm currently trying to implement an Encoder-Decoder architecture for text summarization based on Transformers. Thus I need ti apply MultiHeadAttention on the Decoder site of the model. Since I want ...
2
votes
1answer
101 views
Using Subsequent Mask in Transformer Leads to NaN Outputs
I am trying to implement an autoregressive transformer model similar to the paper attention is all you need. From what I have understood, in order to replicate the architecture fully, I need to give ...
0
votes
0answers
12 views
Can transformers be used to solve for a number of independent polynomial inequalities or polynomial equations?
I'm interested in solving constraint satisfaction problems involving polynomial functions of real variables using transformers. The papers available only deal with boolean SATs in CNF format e.g., ...
0
votes
0answers
11 views
Do sklearn Pipelines automatically split big datasets in chunks for the transform method?
Do sklearn Pipelines automatically split big datasets in chunks for the transform method?
Each transformer in the pipe has a transform method. It seems as sklearn by default pushes all X_train into ...
