Newest 'transformer' Questions

2

votes

1answer

40 views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...

asked yesterday

Pablo Messina

2751 silver badge7 bronze badges

1

vote

1answer

23 views

Why are convolutions still used in some Transformer networks for speech enhancement?

So I’ve read in Attention is All You Need that Transformers remove the need for recurrence and convolutions entirely. However, I’ve seen some TNNs (such as SepFormer, DPTNet, and TSTNN) that still ...

machine-learning transformer attention-mechanism

asked Apr 14 at 1:47

J. Herrera

354 bronze badges

2

votes

1answer

42 views

Does multi-head attention remove the need for self-attention?

The title may be confusing but suppose I were to build Transformer Neural Network with a masking network that utilizes multi-head attention (like that in SepFormer), would adding self-attention in the ...

deep-learning transformer attention-mechanism

asked Apr 12 at 8:52

J. Herrera

354 bronze badges

0

votes

0answers

14 views

How does multi-head attention on “multiple attention axes” works?

I would like to apply an self-attention mechanism on a multichannel audio spectrogram, so a 3D tensor. In the original Transformer paper, self-attention is applied on vector (embedded words) within a ...

deep-learning neural-network transformer attention-mechanism

asked Apr 8 at 8:04

Peehay

112 bronze badges

0

votes

0answers

23 views

Save and Load Simple Transformer Model

I have trained Text classifier using simpleTranformer.ai I am struggling to save and load the model in docker container. Please let me know how can I save the trained model and then load it into ...

python transformer torch

asked Apr 6 at 10:49

SK Singh

112 bronze badges

0

votes

2answers

27 views

Intuition of “Head” in Attention models (Transformer)?

I keep seeing the "head" in attention models (transformers). Aside from the mathematical formula, could anyone please share the intuition behinds the idea "head" ? Thanks a lot!

transformer attention-mechanism

asked Apr 5 at 23:49

Edamame

2,0955 gold badges14 silver badges26 bronze badges

0

votes

0answers

20 views

learn information from text and resolve problem using transformers

Let's imagine that we have some question, like this: "x multiplied by x equals 9. What is x?" For this easy question answer is +-3. I want to make AI model answer on questions like that. To ...

tensorflow transformer question-answering

asked Apr 5 at 9:42

Nikto

717 bronze badges

0

votes

0answers

12 views

Validation Loss Decreases Then Increases to The Same Values

I am training a transformer-based model using Pytorch. The training loss decreases until it hits a floor, which is expected. However, the validation loss decreases to a minimum then starts increasing. ...

machine-learning neural-network transformer validation loss

asked Mar 31 at 20:08

Nyxynyx

1013 bronze badges

0

votes

0answers

9 views

Is the output of this loss function considered as a percentage or something else?

I am currently working on fine-tuning the PEGASUS model for abstractive summarization. The script for fine-tuning the class Trainer from Transformers is imported. The output of training loss is ...

loss-function transformer

asked Mar 31 at 18:49

Karim Fayed

1

0

votes

1answer

19 views

BERT Masked Language Model question

I have been reading about BERT from the internet, and from what I understand the point of masked language modelling for BERT pretraining is so that BERT will learn to guess a "masked" word ...

bert transformer

asked Mar 28 at 15:04

scaraven

1

0

votes

0answers

20 views

Why do relative positional embeddings instead of absolute positional encoding improve the Transformer?

I've been researching the Music Transformer, the paper for which introduced an efficient algorithm to compute Relative Positional Embeddings in a Transformer. I know that Relative Positional ...

machine-learning transformer attention-mechanism

asked Mar 25 at 18:01

spectraldoy

11 bronze badge

0

votes

0answers

105 views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...

machine-learning tensorflow transformer named-entity-recognition huggingface

asked Mar 20 at 1:59

Khachatur Mirijanyan

132 bronze badges

0

votes

1answer

11 views

There could be a problem with the linear layer after the attention inside a transformer?

My question regards this image: It seems that after the multi head attention there is a linear layer as they mention also from here: the linearity is given by the weights W^{o}. my quesion is: for ...

deep-learning neural-network transformer attention-mechanism masking

asked Mar 17 at 14:48

erre4

857 bronze badges

0

votes

1answer

25 views

How does the Transformer predict n steps into the future?

I have barely been able to find an implementation of the Transformer (that is not bloated nor confusing), and the one that I've used as reference was the PyTorch implementation. However, the Pytorch ...

tensorflow pytorch transformer sequence-to-sequence

asked Mar 9 at 14:08

skevelis

32 bronze badges

1

vote

1answer

56 views

Decoder Transformer feedforward

I have a question about the decoder transformer feed forward during training. Let's pick an example: input data "i love the sun" traduction i want to ...

neural-network deep-learning transformer attention-mechanism masking

asked Mar 5 at 10:40

erre4

857 bronze badges

0

votes

0answers

10 views

transformers require a large d_model even when the input cardinality is low?

I'm training a transformer encoder for an NLP task over character data, so the cardinality of my input is 26. I've noticed that if I want to create a strong model, I need make $x$ == my embedding ...

lstm transformer parameter

asked Feb 27 at 11:37

ihadanny

1,1971 gold badge10 silver badges18 bronze badges

0

votes

1answer

105 views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...

nlp word-embeddings bert transformer embeddings

asked Feb 25 at 14:42

Sayali Sonawane

1,6912 gold badges8 silver badges11 bronze badges

1

vote

1answer

33 views

Why does my manual derivative of Layer Normalization imply no gradient flow?

I recently tried computing the derivative of the layer norm function (https://arxiv.org/abs/1607.06450), an essential component of transformers, but the result suggests that no gradient flows through ...

normalization transformer gradient

asked Feb 19 at 21:43

Alex

134 bronze badges

0

votes

1answer

39 views

How to use BERT in seq2seq model? [closed]

I would like to use pretrained BERT as encoder of transformer model. The decoder has the same vocabulary as encoder and I am going to use shared embeddings. But I need ...

neural-network bert transformer transfer-learning sequence-to-sequence

asked Feb 14 at 11:40

Andrey

1033 bronze badges

0

votes

1answer

39 views

Are all 110 million parameter in bert are trainable

I am trying to understand are all these 110 million parameters trainable of bert uncased model. Is there any non trainable parameters in this image below? By trainable I understand they are ...

deep-learning neural-network nlp bert transformer

asked Feb 6 at 5:38

prog

1153 bronze badges

0

votes

1answer

49 views

What are the inputs to the first decoder layer in a Transformer model during the training phase?

I am trying to wrap my head around how the Transformer architecture works. I think I have a decent top-level understanding of the encoder part, sort of how the Key, Query, and Value tensors work in ...

deep-learning transformer

asked Feb 5 at 14:05

djvaroli

1034 bronze badges

1

vote

1answer

129 views

Backpropagation of a transformer

when a transformer model is trained there is linear layer in the end of decoder which i understand is a fully connected neural network. During training of a transformer model when a loss is obtained ...

deep-learning neural-network nlp bert transformer

asked Feb 5 at 13:22

prog

1153 bronze badges

1

vote

1answer

117 views

Unigram tokenizer: how does it work?

I have been trying to understand how the unigram tokenizer works since it is used in the sentencePiece tokenizer that I am planning on using, but I cannot wrap my head around it. I tried to read the ...

nlp transformer tokenization

asked Feb 2 at 13:28

Johncowk

253 bronze badges

-2

votes

1answer

49 views

Transformers understanding

i have i a big trouble. I don't understand transformers. I understand embedding, rnn's, GAN's, even Attention. But i don't understand transformers. Approximately 2 months ago i decided to avoid usage ...

keras nlp transformer

asked Feb 2 at 12:45

Nikto

717 bronze badges

3

votes

2answers

104 views

Layer normalization details in GPT-2

I've read that GPT-2 and other transformers use layer normalization before the self-attention and feedforward blocks, but I am still unsure exactly how the normalization works. Let's say that our ...

deep-learning normalization transformer

asked Jan 27 at 14:24

rampatowl

814 bronze badges

0

votes

1answer

18 views

Transformer architecture question

I am hand-coding a transformer (https://arxiv.org/pdf/1706.03762.pdf) based primarily on the instructions I found at this blog: http://jalammar.github.io/illustrated-transformer/. The first attention ...

neural-network deep-learning nlp transformer attention-mechanism

asked Jan 26 at 15:01

mkohler

233 bronze badges

1

vote

1answer

20 views

Train a final model with the full data

I have trained a few NLP models, measured their performances and now I want to create a final model for production trained with all the data I have available. I'm working in text classification and I'...

pytorch transformer

asked Jan 26 at 13:38

Miguel

1112 bronze badges

2

votes

1answer

52 views

How to evaluate the quality of speech-to-text data without access to the true labels?

I am dealing with a data set of transcribed call center data, where customers are being recorded when interacting with the agent. This is then automatically transcribed by an external transcription ...

nlp text-mining transformer speech-to-text

asked Jan 24 at 1:12

miri_h_ds

213 bronze badges

0

votes

1answer

38 views

How do the linear layers in the attention mechanism work?

I think I now the answer to my question but I dont really get confirmation. When taking a look at the multi-head-attention block as presented in "Attention Is All You Need" we can see that ...

machine-learning nlp transformer attention-mechanism

asked Jan 22 at 11:45

T Piper

1235 bronze badges

0

votes

0answers

29 views

How to determine sentence similarity labels for sentence transformer fine-tuning?

I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...

word-embeddings bert transformer

asked Jan 19 at 9:30

Emerson Hsieh

1012 bronze badges

0

votes

1answer

32 views

why do transformers mask at every layer instead of just at the input layer?

working thru the annotated transformer, I see that every layer in both the encoder (mask paddings) and decoder (mask padding + future positions) get masked. Why couldn't it be simplified to just one ...

neural-network nlp transformer

asked Jan 17 at 20:04

ihadanny

1,1971 gold badge10 silver badges18 bronze badges

1

vote

0answers

24 views

how is the linear relation between positional encoding helping attention?

I'm reading the annotated transformer, and interested in the mechanics behind the positional encoding. I understand the linear relation between position $t$ and position $t+\phi$, and understand that ...

nlp transformer attention-mechanism

asked Jan 16 at 21:00

ihadanny

1,1971 gold badge10 silver badges18 bronze badges

7

votes

2answers

332 views

Transformer model: Why are word embeddings scaled before adding positional encodings?

While going over a Tensorflow tutorial for the Transformer model I realized that their implementation of the Encoder layer (and the Decoder) scales word embeddings by sqrt of embedding dimension ...

tensorflow nlp transformer attention-mechanism

asked Jan 13 at 10:10

Milad Shahidi

2932 silver badges6 bronze badges

1

vote

0answers

66 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...

nlp transformer stanford-nlp tokenization huggingface

asked Jan 13 at 7:02

cerofrais

1113 bronze badges

1

vote

0answers

40 views

Using numpy.ndarray in machine learning sklearn.preprocessing model

I'm having a problem storing and using an array on one model that I'm building in sklearn, Here is what I'm doing: I'm converting an image to numpy and storing as numpy.ndarray in my dataframe (there ...

machine-learning image-classification numpy dataframe transformer

asked Jan 9 at 17:23

Mateus Oliveira Cabral

213 bronze badges

0

votes

1answer

183 views

What is the difference between GPT blocks and BERT blocks

Nowadays many applications only use the Encoder and Decoder part of the Transformer respectively. I am having trouble understanding the difference though. If GPT uses Decoder only and BERT uses ...

bert transformer openai-gpt

asked Jan 7 at 15:43

CD86

1032 bronze badges

0

votes

0answers

27 views

List of Google T5 possible operations

I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...

nlp transfer-learning transformer huggingface

asked Jan 7 at 15:21

tschomacker

11 bronze badge

0

votes

1answer

57 views

What is the number of neurons for the input layer of the BERT?

I think it is the vocab size. However I am not sure and I appreciate your help.

bert transformer

asked Jan 6 at 20:42

Adel

1173 bronze badges

1

vote

0answers

12 views

What does attention weights output from Transformer network do?

I'm trying to understand transformer networks. I want to know that are the attention weights, which are the outputs from forward/predict method where we get final output and attention weights as ...

neural-network transformer

asked Jan 5 at 6:35

Ubhi

114 bronze badges

1

vote

1answer

31 views

Inference order in BERT masking task

In BERT, multiple words in a single sentence can be masked at once. Does the model infer all of those words at once or iterate over them in either left to right or some other order? For example: The ...

neural-network nlp bert transformer language-model

asked Dec 31 '20 at 20:33

Patrick Flynn

132 bronze badges

1

vote

0answers

72 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...

class-imbalance multiclass-classification bert transformer smote

asked Dec 31 '20 at 14:09

nikhil6041

111 bronze badge

0

votes

2answers

34 views

Can Transformer Models be used for Training Chatbots?

Can Transformer Models be used for Training Chatbots? Note - I am talking about the transformer model google released on the paper 'Attention is all you need'

machine-learning deep-learning nlp transformer chatbot

asked Dec 27 '20 at 3:29

Trump

11 bronze badge

0

votes

0answers

25 views

How to use Pytorch's Transformer module “out of the box”

I am working on implementing my first transformer, and recently I've been working in Pytorch and I see that they offer a pre-packaged transformer model. Here are the docs. I have been reading through ...

pytorch transformer

asked Dec 26 '20 at 20:14

jeffery_the_wind

1256 bronze badges

0

votes

1answer

18 views

Many questions training unbalanced and duplicated data

I'm a DS student. I have like 30.000 of bank statements, all labeled with a specific category(cat1, cat2, ...). With that data I'm trying to train a classification model but I found several problems: ...

classification training class-imbalance transformer

asked Dec 18 '20 at 15:10

Jack Fenn

112 bronze badges

0

votes

0answers

26 views

What's the best method to merge N categorical features into one and keep it as categorical

I'm training a Transformer model and it requires one input sentence and N optional labels, not classes cause it's a multi-label and multi-class problem so the unique classes turned into labels. I have ...

categorical-data transformer categorical-encoding

asked Dec 17 '20 at 12:29

user58746

0

votes

1answer

221 views

BERT uses WordPiece, RoBERTa uses BPE

In the original BERT paper, section 'A.2 Pre-training Procedure', it is mentioned: The LM masking is applied after WordPiece tokenization with a uniform masking rate of 15%, and no special ...

bert transfer-learning transformer language-model tokenization

asked Dec 11 '20 at 19:10

Adel

1173 bronze badges

-1

votes

1answer

263 views

What's the right input for gpt-2 in NLP

I'm fine-tuning pre-trained gpt-2 for text summarization. The dataset contains 'text' and 'reference summary'. So my question is how to add special tokens to get the right input format. Currently I'm ...

nlp data-science-model transformer openai-gpt

asked Dec 11 '20 at 17:29

yuqiong11

11

3

votes

1answer

38 views

Trained BERT models perform unpredictably on test set

We are training a BERT model (using the Huggingface library) for a sequence labeling task with six labels: five labels indicate that a token belongs to a class that is interesting to us, and one label ...

nlp bert transformer

asked Dec 11 '20 at 10:23

PeterPaul

333 bronze badges

1

vote

1answer

14 views

Understanding the XLNet model for a concrete case

I'm a data science student, recently I reviewed the XLNet paper and I have a doubt about it: Imagine we are using many categories, let's say 200, can this model has problems reaching a good accuracy (...

machine-learning neural-network classification transformer text-classification

asked Dec 8 '20 at 14:03

Jack Fenn

112 bronze badges

-1

votes

1answer

31 views

In transformers, do you understand why are the Value (V) vectors comes from the encoder? And than normalize with the query (Q) vector? [closed]

In transformers, there is a phase for rasidual connection, where the queries and the output from the attention are add and normalize. Can one please give some advise to the motivation of it? Or maybe ...

nlp transformer spatial-transformer

asked Dec 5 '20 at 17:20

eran

1

Questions tagged [transformer]

Related Tags