Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
1
vote
2answers
17 views

Paraphrasing a sentence and changing the tone of it

I am trying to make a model that is capable of translating a sentence into a new and a better form. I would like the model to change the tone and also give it some character. I am using this in my web ...
0
votes
0answers
30 views

Train a transformer for classification task using time series data

I need to implement and train a Transformer-based model to classify users (binary classification) based on some time-series data. For each user, time-series data are stored as a variable number of ...
0
votes
0answers
9 views

Proper scaling method for time series classification transformer models?

I've asked a similar question about Gradient Boosting Machines already some time ago. This time, I would like to perform time series classifications with a transformer model. I found this Keras ...
0
votes
0answers
37 views

Are there any ways to check default values of pre-trained models before fine-tuning?

Background According to the instruction on Hugging Face page, I'm trying to fine tune pre-trained model for named entity recognition. I think I should try Transfer Learning for the first, but there is ...
0
votes
1answer
17 views

An issue for sub-word tokenization preprocessing transformer

I'm stacked with executing the sub-word tokenization preprocessing to use transformer. According to the tutorial on the article, I have executed the sample code. However, one function was not defined ...
0
votes
1answer
30 views

Sub-word tokenization preprocessing to use transformer

I'm stacked with executing the sub-word tokenization preprocessing to use transformer. According to the tutorial on the article, I have executed the sample code. However, one function was not defined ...
0
votes
0answers
9 views

How to frame queries and answers from customer Agent utterances using Deep Learning SOTA

I am working with smart-reply use case for Async chat customer and ...
0
votes
1answer
57 views

Error to load a pre-trained BERT model

Background I'm reading this article about a natural language task, named entity recognition and trying to load a pre-trained BERT model on Google colaboratory. How can I fix an error to load a pre-...
0
votes
0answers
13 views

Can someone explain Minimum Bayes Risk intuituvely? (Explain like I'm five)

I am learning Transformer and studying Decoding, such as beam search and random sampling which are easy to understand. However, when it comes to Minimum Bayes Risk, it is more difficult. Please help
0
votes
0answers
31 views

How to load fine-tuned Electra (TFElectraForSequenceClassification) Model?

I have fine-tuned an Electra Model using the following code. ...
0
votes
1answer
77 views

time series anomaly detection

I want to ask for time series anomaly detection we can apply tnn on multiple features or not? I used transformer for sentiment analysis where I have to provide a sentence and it predicts its output as ...
0
votes
1answer
50 views

Self-Attention Summation and Loss of Information

In self-attention, the attention for a word is calculated as: $$ A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>} $$ My question is why we sum over the ...
0
votes
0answers
69 views

Getting Word Embeddings for Sentences using long-former model?

I am new to Huggingface and have few basic queries. This post might be helpful to others as well who are starting to use longformer model from huggingface. Objective: Create Sentence/document ...
0
votes
0answers
51 views

Can we add positional encoding to time series input for time series prediction?

I want to use classical machine learning models such XGBoost for my time series prediction. Since the input data for XGBoost/sklearn based models is 2d i.e. (n_samples, n_features), I want to encode ...
0
votes
1answer
34 views

How do the linear + softmax layers give out word probabilities in Transformer network?

I am trying to implement a transformer network from scratch in pytorch to understand it. I am using The illustrated transformer for guidance. The part where I am stuck is about how do we go from the ...
2
votes
1answer
37 views

BERT Optimization for Production

I'm using BERT to transform text into 768 dim vector, It's multilingual : ...
1
vote
2answers
2k views

What is the difference between BERT and Roberta

I want to understand the difference between BERT and Roberta. I saw the article below. https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8 It mentions that ...
0
votes
0answers
12 views

How to prepare data for TpyTorch's 3d attn_mask argument in MultiHeadAttention

I'm currently trying to implement an Encoder-Decoder architecture for text summarization based on Transformers. Thus I need ti apply MultiHeadAttention on the Decoder site of the model. Since I want ...
2
votes
1answer
154 views

Using Subsequent Mask in Transformer Leads to NaN Outputs

I am trying to implement an autoregressive transformer model similar to the paper attention is all you need. From what I have understood, in order to replicate the architecture fully, I need to give ...
0
votes
0answers
14 views

Can transformers be used to solve for a number of independent polynomial inequalities or polynomial equations?

I'm interested in solving constraint satisfaction problems involving polynomial functions of real variables using transformers. The papers available only deal with boolean SATs in CNF format e.g., ...
0
votes
0answers
12 views

Do sklearn Pipelines automatically split big datasets in chunks for the transform method?

Do sklearn Pipelines automatically split big datasets in chunks for the transform method? Each transformer in the pipe has a transform method. It seems as sklearn by default pushes all X_train into ...
0
votes
1answer
42 views

How to write a generator to fine-tune transformer based models (Tensorflow)

I have been trying to write a generator for DistillBertFast model ...
0
votes
0answers
25 views

Why positional embeddings are summed with word embeddings instead of concatenation? [duplicate]

I was trying to understand the architecture of transformers especially for the "Positional encoding" part and more precisely on "Why positional embeddings are summed with word ...
0
votes
2answers
21 views

Attention transformation - matrices

Could somebody explain which matrix dimension should be found here - K? and if it is for example 3X3, should I use just 9?
0
votes
1answer
17 views

Ways to build Abstractive summarisation and what are it's challenges

What are state of art techniques to build Abstractive summarisation on some paragraphs or articles and what kind of hurdles or challenges are there to approach this problem?
0
votes
0answers
61 views

Spatial positional encodings Vs Learned positional encodings(Object queries)

I have been trying to understand facebook's Detection transformer(DeTr) paper. Architecture Most of the explanation about the architecture is straightforward. I don't especially understand the ...
0
votes
1answer
53 views

why multiple attention heads learn differently

In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
0
votes
1answer
42 views

Attention weights - change during learning and prediction

Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values. Please ...
3
votes
0answers
62 views

Struggling to understand/implement Transformer Decoder

I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training. What I have ...
0
votes
0answers
49 views

GPU acceleration with Flask API (BERT Transformer)

I have learned two BERT Transformer models, which both solve the text classification problem. Both artificial networks work well and I don’t see any problem while using them on my GPU and CPU as well ...
1
vote
0answers
9 views

Transformer removes heading labels [closed]

Question: Is there a way to retain/include headers once they go through a make_column_transformer? My dataframe has the usual columns and rows, but a header is included. Example: When I take the data ...
0
votes
0answers
39 views

Does NSP task corrupt context during pre-training?

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
0
votes
0answers
18 views

How gpt 3 handle exploding gradients?

In deep neural networks, sometimes we see the problem of exploding and vanishing gradients. So how is it handled by GPT 3 which has billions of parameters?
0
votes
0answers
6 views

1D Data for NLP models

My dataset( Network traffic dataset where we do binary classification)- Number of features in data is 25 Can we use this kind of dataset in NLP models like transformers? If yes what should be the ...
0
votes
0answers
34 views

An algorithm to extract the purpose of a document

I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
0
votes
0answers
60 views

Transformer: English -> Source Code Training Accuracy stuck 60% and validation 40%

I'm working on my final year project which is to write a model that takes as an input an english sentence and generate source code (currently testing on english->javascript dataset provided by ...
1
vote
0answers
19 views

Why do we need dot product as part of the Transformer's training process?

I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
0
votes
0answers
65 views

Understanding difference between benefits of having multi-head attention and of the process of learning Q,K,V embeddings in a single head

From reading different papers and blogposts I got an understanding that learning embeddings with a single attention head and with multiple heads serves essentially the same purpose. In this blogpost ...
1
vote
1answer
105 views

In Transformer's multi-headed attention, how attending "different representation subspaces at different positions" is achieved?

Question partially inspired by this post about the need of multi-head attention mechanism. For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(...
2
votes
1answer
172 views

What exactly is the linear layer in the transformer model?

Please see this image: There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here: Are these linear layers ...
0
votes
0answers
22 views

Scoring function for transformers (BERT etc)

While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The ...
0
votes
0answers
6 views

What techniques are there to train custom sentence classification models with reasonable memory footprint?

We are currently working on tasks that involve user-inputted data (e.g., question-answers, short-answer-grading), with a framework that will allow them to be improved through active learning. However, ...
0
votes
1answer
28 views

Transformer: where is the output of the last FF sub-layer of the encoder used?

In the "Attention Is All You Need" paper, the decoder consists of two attention sub-layers in each layer followed by a FF sub-layer. The first is a masked self attention which gets as an ...
0
votes
0answers
19 views

Verifying the implementation of Multihead Attention in Transformer

I have implemented the MultiAttention head in Transformers. There are so many implementations around so its confusing. Can ...
0
votes
1answer
107 views

Dimensions of Transformer - dmodel and depth

Trying to understand the dimensions of the Multihead Attention component in Transformer referring the following tutorial https://...
1
vote
1answer
105 views

Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts?

Is this some kind of estimation of the relative positions of words in the training texts? are they creating some kind of statistical "distribution" of words? is "cat" usually 2 or ...
0
votes
1answer
292 views

Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder

According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments: tgt – the sequence to the ...
1
vote
1answer
76 views

Masked Language Modeling on Domain-specific Data

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
1
vote
1answer
165 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
4
votes
3answers
2k views

Is time series forecasting possible with a transformer?

For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then ...