Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
1
vote
0answers
28 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
1
vote
1answer
17 views

How do I implement Dual-encoder model in Pytorch?

I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model. Here the encoder and decoder share the same weights but I am unable to ...
4
votes
2answers
64 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
1
vote
1answer
20 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
1
vote
1answer
53 views

Proper masking in the transformer model

Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
11 views

the library 'transformers' works also with older version of Tensorflow?

i am working with Tensorflow version 1.14 and i would like to use the bert embedding. In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...
0
votes
0answers
21 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
1answer
23 views

In Deep Learning, how many kinds of Attention exist? And what is the history of Attention models? [closed]

How many definitions of attention are commonly employed for Deep Learning tasks? That's what I've encountered up to now: Self-attention Bahdanau Luong Multi-Head (used in Transformers) Could you ...
0
votes
0answers
19 views

Transformer Decoding in Inference mode for Time Series

With the Transformer model from "Attention is all you need" you have to feed in the the actual target during training. However, this can obviously not be done for actual inference. Now usually for ...
0
votes
1answer
28 views

Why seq2seq models are superior to simple LSTMs?

It is common knowledge in the field of Deep Learning that the most powerful Recurrent architecture is the sequence-to-sequence, or seq2seq, for pretty much any task (to time series forecasts, to ...
0
votes
0answers
18 views

How would I define a model that computes “trainable edit distance” \ string similarity for Entity Linking

I want to compute a measure of string similarity based on "edit distance". Classic solutions for edit distance predefine the cost of each editing operations, and use a combination of atomic operations ...
0
votes
0answers
17 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
0
votes
0answers
183 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
0answers
14 views

Input embeddings to Transformers network

I have been learning about Transformer network and most of it clear because of some of the brilliant explanation provided by the experts in the field. Can someone explain about the input word ...
0
votes
0answers
16 views

What will happen if we replace the transformer of BERT to evolved transformer?

If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
0
votes
1answer
62 views

When do you use FunctionTransformer instead of .apply()?

I'm watching a PyData talk from 2017 in which the speaker provides this example for how to use FunctionTransformer for sklearn.preprocessing ...
0
votes
0answers
13 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
0
votes
0answers
39 views

nlp: Translation System: Transformer/GPT2 model: Why do we need to mask future tokens?

I am trying to understand the whole concept of masking the tokens in the transformer/gpt2 model. In this blog post, http://jalammar.github.io/illustrated-gpt2/ the author takes an example where " the ...
2
votes
1answer
54 views

What is auxiliary loss in Character-level Transformer model?

I am reading Character-Level Language Modeling with Deeper Self-Attention from Rami Al-Rfou. In the second page, they had mentioned about Auxiliary Losses which can speed-up the model convergence and ...
0
votes
1answer
65 views

NMT, What if we do not pass input for decoder?

For transformer-based neural machine translation (NMT), take English-Chinese for example, we pass English for encoder and use decoder input(Chinese) attend to encoder output, then final output. What ...
0
votes
0answers
12 views
1
vote
0answers
26 views

How does Byte Pair Encoding work on the byte sequence?

I am reading a paper on OpenAI GPT-2, and in the paper the authors are mentioning that they have performed Byte Pair Encoding (BPE) on the byte sequence themselves, and I am not sure what they meant ...
1
vote
0answers
20 views

Does it make sense to use Transformer encoders on top of a pretrained Word2Vec embedding for a classification task?

As the title says. I am dealing with a text classification task, but I do not have the resources to train a BERT word embedding from scratch. I was thinking of using an existing Word2Vec embedding ...
0
votes
0answers
46 views

Transform data into English, then predict an answer using BERT?

I'm looking for research/examples of closed domain QA systems that utilise pre-trained ML models such as BERT, to perform question-answering on structured data (eg: CSV, JSON) that has been ...
3
votes
1answer
95 views

Bi-directionality in BERT model

I am reading the paper BERT: Pre-training of Deep Bidirectional Transformers for Language Understanding that can be found here. It looks to me that the crux of the paper is using masked inputs to ...
2
votes
0answers
29 views

What is Bit Per Character?

What is Bits per Character (bpc) metric which has been used to measure the model accuracy with reference to text8 and ...
2
votes
0answers
351 views

In a Transformer model, why does one sum positional encoding to the embedding rather than concatenate it?

While reviewing the Transformer architecture, I realized something I didn't expect, which is that : the positional encoding is summed to the word embeddings rather than concatenated to it. ...
0
votes
0answers
31 views

Why gpt-2 could apply to other tasks without fine-tune?

Language Models are Unsupervised Multitask Learners https://github.com/openai/gpt-2
0
votes
1answer
76 views

Transformer for neural machine translation: is it possible to predict each word in the target sentence in a single forward pass?

I want to replicate the Transformer from the paper Attention Is All You Need in PyTorch. My question is about the decoder branch of the Transformer. If I understand correctly, given a sentence in the ...
0
votes
0answers
84 views

training decoder only transformer for text generation

i have been trying to figure out a way to use the decoder for next word prediction tasks (given a sequence of tokens). For this purpose i modified the existing tutorial to ignore encoder inputs in the ...
1
vote
0answers
150 views

Pytorch: How to implement nested transformers: a character-level transformer for words and a word-level transformer for sentences?

I have a model in mind, but I'm having a hard time figuring out how to actually code it in Pytorch, especially when it comes to training the model (e.g. how to define mini-batches, etc.). First of all ...
4
votes
3answers
78 views

Test dataset with categorical variable value not present in train dataset & transformer

I want to replace values of a categorical variable ( named 'six' ) by the mean of my target variable ( named 'target' ). I am fitting a transformer doing just that on a train dataset df and then ...
2
votes
1answer
165 views

Pyspark Pipeline Custom Transformer

I'm having some trouble understanding the creation of custom transformers for Pyspark pipelines. I am writing a custom transformer that will take the dataframe column Company and remove stray commas: ...
2
votes
1answer
1k views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
0
votes
2answers
505 views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
0answers
16 views

Data sharing between Custom Tranformer and estimator in pyspark

Using Titanic Dataset for classification. Created custom Estimator and transformer for pipeline. some of logical operation need to be done in fit() method and data ...
3
votes
2answers
59 views

Problem trying to build my own sklean transformer

I build the following sklearn transformer : ...
16
votes
2answers
9k views

What is the positional encoding in the transformer model?

I'm new to ML and this is my first question here, so sorry if my question is silly. I'm trying to read and understand the paper Attention is all you need and in it, there is a picture: I don't know ...
0
votes
1answer
301 views

Which is better: GPT or RelGAN for text generation?

Based on my understanding, gpt or gpt-2 are using language model loss to train and generate text, which do not contains GAN. So which is better: GPT vs RelGAN/LeakGAN/SeqGAN/TextGAN I am so ...
2
votes
0answers
32 views

How to prepare the data for text generation task

First, I'm not sure whether the model contains the encoder during training. EOS means end-of-sentence. Encoder and decoder are part of transformer network. If without-encoder, training time: ...
1
vote
1answer
37 views

The principle of LM deep model

Language model(LM) is the task of predicting the next word. Does the deep model need the encoder? From the ptb code of tensor2tensor, I find the deep model do not contains the encoder. Or both with-...
2
votes
1answer
755 views

Incrementally Train BERT with minimum QnA records - to get improved results

We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/...
2
votes
0answers
521 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
9
votes
1answer
3k views

Can BERT do the next-word-predict task?

As BERT is bidirectional (uses bi-directional transformer), is it possible to use it for the next-word-predict task? If yes, what needs to be tweaked?
0
votes
1answer
242 views

What is the reason for the speedup of transformer-xl?

The inference speed of transformer-xl is faster than transformer. Why? If state reuse is the reason, so it is compared by two 32seq_len + state-reuse vs one 64seq_len + no-state-reuse?