Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
45
questions
1
vote
0answers
28 views
Why do BERT classification do worse with longer sequence length?
I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
1
vote
1answer
17 views
How do I implement Dual-encoder model in Pytorch?
I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model.
Here the encoder and decoder share the same weights but I am unable to ...
4
votes
2answers
64 views
Why Decision Tree Classifier is not working with categorical value?
I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx.
I am trying to model a Decision Tree Classifier as part of an ...
1
vote
1answer
20 views
Measuring quality of answers from QnA systems
I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
1
vote
1answer
53 views
Proper masking in the transformer model
Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
11 views
the library 'transformers' works also with older version of Tensorflow?
i am working with Tensorflow version 1.14 and i would like to use the bert embedding.
In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...
0
votes
0answers
21 views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
23 views
In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]
How many definitions of attention are commonly employed for Deep Learning tasks?
That's what I've encountered up to now:
Self-attention
Bahdanau
Luong
Multi-Head (used in Transformers)
Could you ...
0
votes
0answers
19 views
Transformer Decoding in Inference mode for Time Series
With the Transformer model from "Attention is all you need" you have to feed in the the actual target during training. However, this can obviously not be done for actual inference. Now usually for ...
0
votes
1answer
28 views
Why seq2seq models are superior to simple LSTMs?
It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to ...
0
votes
0answers
18 views
How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking
I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
0
votes
0answers
17 views
BERT for non-textual sequence data
I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
0
votes
0answers
183 views
Preprocessing for Text Classification in Transformer Models (BERT variants)
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a W2V or Glove, ...
0
votes
0answers
14 views
Input embeddings to Transformers network
I have been learning about Transformer network and most of it clear because of some of the brilliant explanation provided by the experts in the field.
Can someone explain about the input word ...
0
votes
0answers
16 views
What will happen if we replace the transformer of BERT to evolved transformer?
If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
0
votes
1answer
62 views
When do you use FunctionTransformer instead of .apply()?
I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing
...
0
votes
0answers
13 views
What's the best way to store BERT training data (input IDs)
The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
0
votes
0answers
39 views
nlp: Translation System: Transformer/GPT2 model: Why do we need to mask future tokens?
I am trying to understand the whole concept of masking the tokens in the transformer/gpt2 model. In this blog post, http://jalammar.github.io/illustrated-gpt2/ the author takes an example where " the ...
2
votes
1answer
54 views
What is auxiliary loss in Character-level Transformer model?
I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and ...
0
votes
1answer
65 views
NMT, What if we do not pass input for decoder?
For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output.
What ...
0
votes
0answers
12 views
How to understand the difference between query and memory in the attention of transformer?
The code is like this:
...
1
vote
0answers
26 views
How does Byte Pair Encoding work on the byte sequence?
I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant ...
1
vote
0answers
20 views
Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?
As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch.
I was thinking of using an existing Word2Vec embedding ...
0
votes
0answers
46 views
Transform data into English, then predict an answer using BERT?
I'm looking for research/examples of closed domain QA systems that utilise pre-trained ML models such as BERT, to perform question-answering on structured data (eg: CSV, JSON) that has been ...
3
votes
1answer
95 views
Bi-directionality in BERT model
I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.
It looks to me that the crux of the paper is using masked inputs to ...
2
votes
0answers
29 views
What is Bit Per Character?
What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
2
votes
0answers
351 views
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
While reviewing the Transformer architecture, I realized something I didn't expect, which is that :
the positional encoding is summed to the word embeddings
rather than concatenated to it.
...
0
votes
0answers
31 views
Why gpt-2 could apply to other tasks without fine-tune?
Language Models are Unsupervised Multitask Learners
https://github.com/openai/gpt-2
0
votes
1answer
76 views
Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?
I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the ...
0
votes
0answers
84 views
training decoder only transformer for text generation
i have been trying to figure out a way to use the decoder for next word prediction tasks (given a sequence of tokens). For this purpose i modified the existing tutorial to ignore encoder inputs in the ...
1
vote
0answers
150 views
Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?
I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all ...
4
votes
3answers
78 views
Test dataset with categorical variable value not present in train dataset & transformer
I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).
I am fitting a transformer doing just that on a train dataset df and then ...
2
votes
1answer
165 views
Pyspark Pipeline Custom Transformer
I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.
I am writing a custom transformer that will take the dataframe column Company and remove stray commas:
...
2
votes
1answer
1k views
what is the first input to the decoder in a transformer model?
The image is from url: Jay Alammar on transformers
K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
0
votes
2answers
505 views
What is the use of [SEP] in paper BERT?
I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun.
However, I have a question.
If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
0answers
16 views
Data sharing between Custom Tranformer and estimator in pyspark
Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
3
votes
2answers
59 views
16
votes
2answers
9k views
What is the positional encoding in the transformer model?
I'm new to ML and this is my first question here, so sorry if my question is silly.
I'm trying to read and understand the paper Attention is all you need and in it, there is a picture:
I don't know ...
0
votes
1answer
301 views
Which is better: GPT or RelGAN for text generation?
Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN.
So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN
I am so ...
2
votes
0answers
32 views
How to prepare the data for text generation task
First, I'm not sure whether the model contains the encoder during training.
EOS means end-of-sentence. Encoder and decoder are part of transformer network.
If without-encoder, training time:
...
1
vote
1answer
37 views
The principle of LM deep model
Language model(LM) is the task of predicting the next word.
Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder.
Or both with-...
2
votes
1answer
755 views
Incrementally Train BERT with minimum QnA records - to get improved results
We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/...
2
votes
0answers
521 views
How Transformer is Bidirectional - Machine Learning
Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
9
votes
1answer
3k views
Can BERT do the next-word-predict task?
As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
0
votes
1answer
242 views
What is the reason for the speedup of transformer-xl?
The inference speed of transformer-xl is faster than transformer.
Why?
If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?
