Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
193
questions
0
votes
1answer
28 views
How to Transform Rows into columns & Fill the values with Existing Values using Pandas as DataFrame
Here, I would like to transform rows into new columns and fill the column values with specific rows.
Sample input:
0
votes
0answers
16 views
Self-Attention Summation and Loss of Information
In self-attention, the attention for a word is calculated as:
$$
A(q, K, V) = \sum_{i} \frac{exp(q.k^{<i>})}{\sum_{j} exp(q.k^{<j>})}v^{<i>}
$$
My question is why we sum over the ...
0
votes
0answers
20 views
Getting Word Embeddings for Sentences using long-former model?
I am new to Huggingface and have few basic queries. This post might be helpful to others as well who are starting to use longformer model from huggingface.
Objective:
Create Sentence/document ...
0
votes
0answers
19 views
Can we add positional encoding to time series input for time series prediction?
I want to use classical machine learning models such XGBoost for my time series prediction. Since the input data for XGBoost/sklearn based models is 2d i.e. (n_samples, n_features), I want to encode ...
0
votes
1answer
22 views
How do the linear + softmax layers give out word probabilities in Transformer network?
I am trying to implement a transformer network from scratch in pytorch to understand it. I am using The illustrated transformer for guidance. The part where I am stuck is about how do we go from the ...
2
votes
1answer
26 views
BERT Optimization for Production
I'm using BERT to transform text into 768 dim vector, It's multilingual :
...
1
vote
2answers
70 views
What is the difference between BERT and Roberta
I want to understand the difference between BERT and Roberta. I saw the article below.
https://towardsdatascience.com/bert-roberta-distilbert-xlnet-which-one-to-use-3d5ab82ba5f8
It mentions that ...
0
votes
0answers
7 views
How to prepare data for TpyTorch's 3d attn_mask argument in MultiHeadAttention
I'm currently trying to implement an Encoder-Decoder architecture for text summarization based on Transformers. Thus I need ti apply MultiHeadAttention on the Decoder site of the model. Since I want ...
2
votes
1answer
23 views
Using Subsequent Mask in Transformer Leads to NaN Outputs
I’m trying to implement an autoregressive transformer model similar to the paper “attention is all you need”. From what I’ve understood, in order to replicate the architecture fully, I need to give ...
0
votes
0answers
12 views
Can transformers be used to solve for a number of independent polynomial inequalities or polynomial equations?
I'm interested in solving constraint satisfaction problems involving polynomial functions of real variables using transformers. The papers available only deal with boolean SATs in CNF format e.g., ...
0
votes
0answers
10 views
Do sklearn Pipelines automatically split big datasets in chunks for the transform method?
Do sklearn Pipelines automatically split big datasets in chunks for the transform method?
Each transformer in the pipe has a transform method. It seems as sklearn by default pushes all X_train into ...
0
votes
1answer
13 views
How to write a generator to fine-tune transformer based models (Tensorflow)
I have been trying to write a generator for DistillBertFast model
...
0
votes
0answers
14 views
Why positional embeddings are summed with word embeddings instead of concatenation? [duplicate]
I was trying to understand the architecture of transformers especially for the "Positional encoding" part and more precisely on "Why positional embeddings are summed with word ...
0
votes
2answers
16 views
Attention transformation - matrices
Could somebody explain which matrix dimension should be found here - K? and if it is for example 3X3, should I use just 9?
0
votes
1answer
15 views
Ways to build Abstractive summarisation and what are it's challenges
What are state of art techniques to build Abstractive summarisation on some paragraphs or articles and what kind of hurdles or challenges are there to approach this problem?
0
votes
0answers
18 views
Spatial positional encodings Vs Learned positional encodings(Object queries)
I have been trying to understand facebook's Detection transformer(DeTr) paper.
Architecture
Most of the explanation about the architecture is straightforward.
I don't especially understand the ...
0
votes
1answer
42 views
why multiple attention heads learn differently
In transformer architecture multi head attention blocks are used. While visualizing their output it can be seen that every layer has learnt different relations of words. e.g., layer 5 has learnt that &...
0
votes
1answer
30 views
Attention weights - change during learning and prediction
Assume a simple LSTM Followed by Attention layer or a full transformer architecture. The attention weights are learnt during training, which get multiplied with keys, queries and values.
Please ...
3
votes
0answers
28 views
Struggling to understand/implement Transformer Decoder
I'm struggling to understand the decoder in a Transformer model, specifically with regards to some aspects of its architecture as well as how it actually handles the data during training.
What I have ...
0
votes
0answers
18 views
GPU acceleration with Flask API (BERT Transformer)
I have learned two BERT Transformer models, which both solve the text classification problem. Both artificial networks work well and I don’t see any problem while using them on my GPU and CPU as well ...
1
vote
0answers
8 views
Transformer removes heading labels [closed]
Question: Is there a way to retain/include headers once they go through a make_column_transformer?
My dataframe has the usual columns and rows, but a header is included.
Example:
When I take the data ...
0
votes
0answers
27 views
Does NSP task corrupt context during pre-training?
During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
0
votes
0answers
11 views
How gpt 3 handle exploding gradients?
In deep neural networks, sometimes we see the problem of exploding and vanishing gradients. So how is it handled by GPT 3 which has billions of parameters?
0
votes
0answers
6 views
1D Data for NLP models
My dataset( Network traffic dataset where we do binary classification)-
Number of features in data is 25
Can we use this kind of dataset in NLP models like transformers? If yes what should be the ...
0
votes
0answers
33 views
An algorithm to extract the purpose of a document
I want to build an algorithm to extract the purpose of the document (scientific papers for example) by extracting the sentences that state the purpose. I don't have many annotated data so I might use ...
0
votes
0answers
39 views
Transformer: English -> Source Code Training Accuracy stuck 60% and validation 40%
I'm working on my final year project which is to write a model that takes as an input an english sentence and generate source code (currently testing on english->javascript dataset provided by ...
1
vote
0answers
18 views
Why do we need dot product as part of the Transformer's training process?
I do understand that dot product conveys the meaning of similarity in a vector space. At the same time it looks like during the training process we are learning the weights( or how much attention) ...
0
votes
0answers
19 views
Understanding difference between benefits of having multi-head attention and of the process of learning Q,K,V embeddings in a single head
From reading different papers and blogposts I got an understanding that learning embeddings with a single attention head and with multiple heads serves essentially the same purpose.
In this blogpost ...
1
vote
1answer
53 views
In Transformer's multi-headed attention, how attending “different representation subspaces at different positions” is achieved?
Question partially inspired by this post about the need of multi-head attention mechanism.
For me though it is still not clear how we will be able to initialise those attention heads in a diverse way(...
2
votes
1answer
45 views
What exactly is the linear layer in the transformer model?
Please see this image:
There are linear layers to modify the Query, key and value matrices and one linear layer after the multi head attention as they mention also from here:
Are these linear layers ...
0
votes
0answers
18 views
Scoring function for transformers (BERT etc)
While using BERT / transformers for NLP tasks, a major problem faced by us was to detect if the answer returned by model is correct or not, or what is the confidence level of the answer. The ...
0
votes
0answers
5 views
What techniques are there to train custom sentence classification models with reasonable memory footprint?
We are currently working on tasks that involve user-inputted data (e.g., question-answers, short-answer-grading), with a framework that will allow them to be improved through active learning. However, ...
0
votes
1answer
21 views
Transformer: where is the output of the last FF sub-layer of the encoder used?
In the "Attention Is All You Need" paper, the decoder consists of two attention sub-layers in each layer followed by a FF sub-layer.
The first is a masked self attention which gets as an ...
0
votes
0answers
15 views
Verifying the implementation of Multihead Attention in Transformer
I have implemented the MultiAttention head in Transformers. There are so many implementations around so its confusing. Can ...
0
votes
1answer
37 views
Dimensions of Transformer - dmodel and depth
Trying to understand the dimensions of the Multihead Attention component in Transformer referring the following tutorial https://...
1
vote
1answer
44 views
Is positional encoding (in transformers) an estimation of the relative positions of words in the training corpus texts?
Is this some kind of estimation of the relative positions of words in the training texts? are they creating some kind of statistical "distribution" of words? is "cat" usually 2 or ...
0
votes
1answer
100 views
Pytorch: understanding the purpose of each argument in the forward function of nn.TransformerDecoder
According to https://pytorch.org/docs/stable/generated/torch.nn.TransformerDecoder.html, the forward function of nn.TransformerDecoder contemplates the following arguments:
tgt – the sequence to the ...
1
vote
1answer
39 views
Masked Language Modeling on Domain-specific Data
My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
0
votes
1answer
68 views
Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)
I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this:
...
1
vote
2answers
309 views
Is time series forecasting possible with a transformer?
For my bachelor project I've been tasked with making a transformer that can forecast time series data, specifically powergrid data. I need to take a univariate time series of length N, that can then ...
3
votes
1answer
150 views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
1
vote
1answer
40 views
Why are convolutions still used in some Transformer networks for speech enhancement?
So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...
2
votes
1answer
80 views
Does multi-head attention remove the need for self-attention?
The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...
0
votes
0answers
30 views
How does multi-head attention on “multiple attention axes” works?
I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a ...
0
votes
2answers
286 views
Save and Load Simple Transformer Model
I have trained Text classifier using simpleTranformer.ai I am struggling to save and load the model in docker container. Please let me know how can I save the trained model and then load it into ...
0
votes
2answers
35 views
Intuition of “Head” in Attention models (Transformer)?
I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!
0
votes
0answers
26 views
learn information from text and resolve problem using transformers
Let's imagine that we have some question, like this: "x multiplied by x equals 9. What is x?"
For this easy question answer is +-3. I want to make AI model answer on questions like that.
To ...
0
votes
0answers
21 views
Validation Loss Decreases Then Increases to The Same Values
I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing.
...
0
votes
0answers
46 views
Is the output of this loss function considered as a percentage or something else?
I am currently working on fine-tuning the PEGASUS model for abstractive summarization.
The script for fine-tuning the class Trainer from Transformers is imported.
The output of training loss is ...
0
votes
1answer
22 views
BERT Masked Language Model question
I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word ...
