Questions tagged [transformer]
Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.
62
questions
2
votes
1answer
25 views
Custom functions and pipelines
I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines.
Situation: I want to fill some missing values with the mean but using groups based on ...
0
votes
0answers
9 views
What is “position” in CNN (im2latex) for Positional Encoding?
I'm trying to build a model that maps images of math
formulas into LaTeX markup.
I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder ...
0
votes
1answer
14 views
In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?
In addition, why do we need a FFN in each layer when we already have attention?
Here's a screenshot of the relevant section from Vaswani et al. (2017):
0
votes
0answers
14 views
Transformers and BERT: dealing with possessives and apostrophes when encode
Let's consider two sentences:
"why isn't Alex's text tokenizing? The house on the left is the Smiths' house"
Now let's tokenize and decode:
...
0
votes
0answers
12 views
How to detokenize a BertTokenizer output?
For example, let's tokenize a sentece "why isn't Alex' text tokenizing":
...
0
votes
2answers
38 views
Does the transformer decoder reuse previous tokens' intermediate states like GPT2?
I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point :
He explains that the decoder of GPT-2 processes input ...
0
votes
1answer
13 views
Transformer-XL architecture
I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, ...
0
votes
1answer
21 views
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
4
votes
2answers
73 views
Why does vanilla transformer has fixed-length input?
I know that in the math on which the transformer is based there is no restriction on the length of input. But I still canāt understand why we should fix it in the frameworks (PyTorch). Because of this ...
0
votes
0answers
18 views
Seeking your advice on XLM-R for NMT
I want to use XLM-R for neural machine translation b/n the same low resource language?
For example: Input-> He sells food(in Catalan) Output-> Food he sells(in Catalan)
Anyone having code example/...
0
votes
1answer
52 views
How do Bahdanau - Luong Attentions use Query, Value, Key vectors?
In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
1
vote
1answer
19 views
Should weight distribution change more when fine-tuning transformers-based classifier?
I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation:
...
2
votes
1answer
49 views
Why does the transformer positional encoding use both sine and cosine?
In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed.
I am wondering why it needs to use both sine and cosine though instead of just ...
0
votes
2answers
37 views
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
1
vote
1answer
25 views
What is the feedforward network in a transformer trained on?
After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns.
What I learned ...
0
votes
0answers
21 views
Pretrained Models for Keyword-Based Text Generation
I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2).
An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
0
votes
0answers
63 views
Cross layer parameter sharing in ALBERT Model
I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross layer parameter sharing to improve the model performance. I don't understand how ...
1
vote
0answers
45 views
Why do BERT classification do worse with longer sequence length?
I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
1
vote
1answer
32 views
How do I implement Dual-encoder model in Pytorch?
I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model.
Here the encoder and decoder share the same weights but I am unable to ...
4
votes
2answers
73 views
Why Decision Tree Classifier is not working with categorical value?
I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx.
I am trying to model a Decision Tree Classifier as part of an ...
1
vote
1answer
23 views
Measuring quality of answers from QnA systems
I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
1
vote
1answer
278 views
Proper masking in the transformer model
Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
13 views
the library 'transformers' works also with older version of Tensorflow?
i am working with Tensorflow version 1.14 and i would like to use the bert embedding.
In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...
1
vote
0answers
44 views
Weight matrices in transformers
I am trying to understand the transformer architecture.
I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
26 views
In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]
How many definitions of attention are commonly employed for Deep Learning tasks?
That's what I've encountered up to now:
Self-attention
Bahdanau
Luong
Multi-Head (used in Transformers)
Could you ...
0
votes
0answers
38 views
Transformer Decoding in Inference mode for Time Series
With the Transformer model from "Attention is all you need" you have to feed in the the actual target during training. However, this can obviously not be done for actual inference. Now usually for ...
0
votes
1answer
45 views
Why seq2seq models are superior to simple LSTMs?
It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to ...
0
votes
0answers
19 views
How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking
I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
0
votes
0answers
20 views
BERT for non-textual sequence data
I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
0answers
653 views
Preprocessing for Text Classification in Transformer Models (BERT variants)
This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models?
I remember for training a W2V or Glove, ...
0
votes
0answers
36 views
Input embeddings to Transformers network
I have been learning about Transformer network and most of it clear because of some of the brilliant explanation provided by the experts in the field.
Can someone explain about the input word ...
0
votes
0answers
18 views
What will happen if we replace the transformer of BERT to evolved transformer?
If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
0
votes
1answer
128 views
When do you use FunctionTransformer instead of .apply()?
I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing
...
0
votes
0answers
13 views
What's the best way to store BERT training data (input IDs)
The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
0
votes
0answers
68 views
nlp: Translation System: Transformer/GPT2 model: Why do we need to mask future tokens?
I am trying to understand the whole concept of masking the tokens in the transformer/gpt2 model. In this blog post, http://jalammar.github.io/illustrated-gpt2/ the author takes an example where " the ...
2
votes
1answer
80 views
What is auxiliary loss in Character-level Transformer model?
I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and ...
0
votes
1answer
93 views
NMT, What if we do not pass input for decoder?
For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output.
What ...
0
votes
0answers
13 views
How to understand the difference between query and memory in the attention of transformer?
The code is like this:
...
1
vote
0answers
39 views
How does Byte Pair Encoding work on the byte sequence?
I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant ...
2
votes
0answers
23 views
Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?
As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch.
I was thinking of using an existing Word2Vec embedding ...
0
votes
0answers
58 views
Transform data into English, then predict an answer using BERT?
I'm looking for research/examples of closed domain QA systems that utilise pre-trained ML models such as BERT, to perform question-answering on structured data (eg: CSV, JSON) that has been ...
3
votes
1answer
140 views
Bi-directionality in BERT model
I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here.
It looks to me that the crux of the paper is using masked inputs to ...
2
votes
0answers
45 views
What is Bit Per Character?
What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
3
votes
0answers
486 views
In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?
While reviewing the Transformer architecture, I realized something I didn't expect, which is that :
the positional encoding is summed to the word embeddings
rather than concatenated to it.
...
0
votes
0answers
34 views
Why gpt-2 could apply to other tasks without fine-tune?
Language Models are Unsupervised Multitask Learners
https://github.com/openai/gpt-2
0
votes
1answer
102 views
Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?
I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the ...
0
votes
0answers
104 views
training decoder only transformer for text generation
i have been trying to figure out a way to use the decoder for next word prediction tasks (given a sequence of tokens). For this purpose i modified the existing tutorial to ignore encoder inputs in the ...
1
vote
0answers
174 views
Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?
I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all ...
4
votes
3answers
81 views
Test dataset with categorical variable value not present in train dataset & transformer
I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ).
I am fitting a transformer doing just that on a train dataset df and then ...
2
votes
1answer
219 views
Pyspark Pipeline Custom Transformer
I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines.
I am writing a custom transformer that will take the dataframe column Company and remove stray commas:
...
