Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
2
votes
1answer
25 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
0
votes
0answers
9 views

What is “position” in CNN (im2latex) for Positional Encoding?

I'm trying to build a model that maps images of math formulas into LaTeX markup. I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder ...
0
votes
1answer
14 views

In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?

In addition, why do we need a FFN in each layer when we already have attention? Here's a screenshot of the relevant section from Vaswani et al. (2017):
0
votes
0answers
14 views

Transformers and BERT: dealing with possessives and apostrophes when encode

Let's consider two sentences: "why isn't Alex's text tokenizing? The house on the left is the Smiths' house" Now let's tokenize and decode: ...
0
votes
0answers
12 views

How to detokenize a BertTokenizer output?

For example, let's tokenize a sentece "why isn't Alex' text tokenizing": ...
0
votes
2answers
38 views

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point : He explains that the decoder of GPT-2 processes input ...
0
votes
1answer
13 views

Transformer-XL architecture

I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, ...
0
votes
1answer
21 views

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
4
votes
2answers
73 views

Why does vanilla transformer has fixed-length input?

I know that in the math on which the transformer is based there is no restriction on the length of input. But I still can’t understand why we should fix it in the frameworks (PyTorch). Because of this ...
0
votes
0answers
18 views

Seeking your advice on XLM-R for NMT

I want to use XLM-R for neural machine translation b/n the same low resource language? For example: Input-> He sells food(in Catalan) Output-> Food he sells(in Catalan) Anyone having code example/...
0
votes
1answer
52 views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
1
vote
1answer
19 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
2
votes
1answer
49 views

Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed. I am wondering why it needs to use both sine and cosine though instead of just ...
0
votes
2answers
37 views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
1
vote
1answer
25 views

What is the feedforward network in a transformer trained on?

After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns. What I learned ...
0
votes
0answers
21 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
0
votes
0answers
63 views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross layer parameter sharing to improve the model performance. I don't understand how ...
1
vote
0answers
45 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
1
vote
1answer
32 views

How do I implement Dual-encoder model in Pytorch?

I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model. Here the encoder and decoder share the same weights but I am unable to ...
4
votes
2answers
73 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
1
vote
1answer
23 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
1
vote
1answer
278 views

Proper masking in the transformer model

Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
13 views

the library 'transformers' works also with older version of Tensorflow?

i am working with Tensorflow version 1.14 and i would like to use the bert embedding. In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...
1
vote
0answers
44 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
26 views

In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]

How many definitions of attention are commonly employed for Deep Learning tasks? That's what I've encountered up to now: Self-attention Bahdanau Luong Multi-Head (used in Transformers) Could you ...
0
votes
0answers
38 views

Transformer Decoding in Inference mode for Time Series

With the Transformer model from "Attention is all you need" you have to feed in the the actual target during training. However, this can obviously not be done for actual inference. Now usually for ...
0
votes
1answer
45 views

Why seq2seq models are superior to simple LSTMs?

It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to ...
0
votes
0answers
19 views

How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking

I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
0
votes
0answers
20 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
0answers
653 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
0answers
36 views

Input embeddings to Transformers network

I have been learning about Transformer network and most of it clear because of some of the brilliant explanation provided by the experts in the field. Can someone explain about the input word ...
0
votes
0answers
18 views

What will happen if we replace the transformer of BERT to evolved transformer?

If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
0
votes
1answer
128 views

When do you use FunctionTransformer instead of .apply()?

I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing ...
0
votes
0answers
13 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
0
votes
0answers
68 views

nlp: Translation System: Transformer/GPT2 model: Why do we need to mask future tokens?

I am trying to understand the whole concept of masking the tokens in the transformer/gpt2 model. In this blog post, http://jalammar.github.io/illustrated-gpt2/ the author takes an example where " the ...
2
votes
1answer
80 views

What is auxiliary loss in Character-level Transformer model?

I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and ...
0
votes
1answer
93 views

NMT, What if we do not pass input for decoder?

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output. What ...
0
votes
0answers
13 views
1
vote
0answers
39 views

How does Byte Pair Encoding work on the byte sequence?

I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant ...
2
votes
0answers
23 views

Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?

As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch. I was thinking of using an existing Word2Vec embedding ...
0
votes
0answers
58 views

Transform data into English, then predict an answer using BERT?

I'm looking for research/examples of closed domain QA systems that utilise pre-trained ML models such as BERT, to perform question-answering on structured data (eg: CSV, JSON) that has been ...
3
votes
1answer
140 views

Bi-directionality in BERT model

I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here. It looks to me that the crux of the paper is using masked inputs to ...
2
votes
0answers
45 views

What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
3
votes
0answers
486 views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
0
votes
0answers
34 views

Why gpt-2 could apply to other tasks without fine-tune?

Language Models are Unsupervised Multitask Learners https://github.com/openai/gpt-2
0
votes
1answer
102 views

Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?

I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the ...
0
votes
0answers
104 views

training decoder only transformer for text generation

i have been trying to figure out a way to use the decoder for next word prediction tasks (given a sequence of tokens). For this purpose i modified the existing tutorial to ignore encoder inputs in the ...
1
vote
0answers
174 views

Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all ...
4
votes
3answers
81 views

Test dataset with categorical variable value not present in train dataset & transformer

I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ). I am fitting a transformer doing just that on a train dataset df and then ...
2
votes
1answer
219 views

Pyspark Pipeline Custom Transformer

I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines. I am writing a custom transformer that will take the dataframe column Company and remove stray commas: ...