Data Science

Questions tagged [transformer]

Ask Question

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

87 questions
Newest
Active
Bountied
Unanswered
Filter by
Sorted by
Tagged with
0
votes
0answers
11 views

BERT reasoning capabilities

I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://...
0
votes
1answer
13 views

what is the difference between positional vector and attention vector used in transformer model?

what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
-1
votes
0answers
8 views

Using feature embeddings of the output of a transformers to represent probabilities of categorical data [closed]

I was considering using a transformer, on input data which can be represented as an embedding, so I can use the attention mechanism in the transformer architecture. As my data is of variable input and ...
0
votes
1answer
18 views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
2answers
36 views

Next sentence prediction in RoBERTa

I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments ...
0
votes
0answers
17 views

What is the difference between register_buffer() and parameter.detach() in PyTorch?

I am writing a PositionalEmbedding() module which is an implementation based on "Attention Is All You Need" using PyTorch. According to the paper, there ...
0
votes
0answers
15 views

Calculating key and value vector in the Transformer's decoder block

I am implementing the transformer model in Pytorch by following Jay Alammar's post and the implementation here. My question is regarding the input to the decoder layer. As shown in the diagram above, ...
0
votes
0answers
12 views

TF classes missing in huggingface's transformers

I've downloaded huggingface's transformers module, but I don't see any of the classes with the "TF" prefixes listed in the documentation such as "TFRobertaForSequenceClassification"...
-1
votes
0answers
6 views

What's the general procedure to include conversational context in binary classification using BERT?

My data set looks something like this: Post | Comment1 | Comment2 | Comment3 | label [0 or 1] Where the 'label' indicates whether 'Comment3' is positive (1) or negative (0) So currently, my input ...
0
votes
0answers
14 views

What are the simplest methods for the label noise problem?

If I have enough low quality data from unsupervised methods or rule-based methods. I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex ...
0
votes
1answer
34 views

How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
0
votes
0answers
44 views

BPE vs WordPiece Tokenization - when to use / which?

What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a ...
0
votes
1answer
30 views

Data quantity is not low but data quality is low, what are the best practices now?

Text classification task, if data quantity is low but data quality is not low. We could use data augment methods for improvement. But the situation is that data quantity is not low and data quality ...
1
vote
1answer
19 views

TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

The input function code: ...
0
votes
0answers
11 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
2
votes
1answer
44 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
1
vote
1answer
35 views

Transformer-based architectures for regression tasks

As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
2
votes
1answer
29 views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
0
votes
1answer
20 views

sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

I need to process some dataframe columns in different steps using ColumnTransformer. The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to ...
3
votes
1answer
219 views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
0
votes
2answers
62 views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
0
votes
0answers
44 views

BERT Implementaion for Sequence Classification

I am trying to implement BERT using HuggingFace - transformers implementation. I am following two links: by analytics-vidhya and by HuggingFace Below is the code: ...
0
votes
0answers
35 views

HuggingFace/Transformers Implementation for Classification

I am trying to implement BERT using HuggingFace - transformers implementation. I am following two links: by analytics-vidhya and by HuggingFace If we consider inputs for both the implementations: 1) ...
0
votes
0answers
35 views

Positional Encoding of Categorical Features in a Time Series Transformer

I am training a Transformer for Multivariate Time Series prediction. I am working with Categorical features and I am thinking of using Positional Encoding to encode them instead of Embedding. Has ...
1
vote
1answer
195 views

Overfitting with text classification using Transformers

I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good ...
1
vote
1answer
57 views

Can BERT be used for predicting words?

I have a question regarding the pre-training section (in particular, the Masked Language Model). In the example Let's stick to improvisation in this skit, by masking the word improvisation, after ...
1
vote
2answers
27 views

What are good toy problems for testing Transformer architectures?

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer ...
2
votes
1answer
44 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
0
votes
0answers
45 views

What is “position” in CNN (im2latex) for Positional Encoding?

I'm trying to build a model that maps images of math formulas into LaTeX markup. I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder ...
1
vote
2answers
30 views

In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?

In addition, why do we need a FFN in each layer when we already have attention? Here's a screenshot of the relevant section from Vaswani et al. (2017):
0
votes
0answers
22 views

Transformers and BERT: dealing with possessives and apostrophes when encode

Let's consider two sentences: "why isn't Alex's text tokenizing? The house on the left is the Smiths' house" Now let's tokenize and decode: ...
0
votes
0answers
72 views

How to detokenize a BertTokenizer output?

For example, let's tokenize a sentece "why isn't Alex' text tokenizing": ...
0
votes
2answers
102 views

Does the transformer decoder reuse previous tokens' intermediate states like GPT2?

I recently read Jay Alammar's blogpost about GPT-2 (http://jalammar.github.io/illustrated-gpt2/) which I found quite clear appart from one point : He explains that the decoder of GPT-2 processes input ...
0
votes
1answer
39 views

Transformer-XL architecture

I am a bit perplex from the transformer-XL architecture that is claimed to solve the issue of context fragmantation. I probably understood it wrong but it looks like all the transformer-XL is doing, ...
0
votes
1answer
89 views

Transformer seq2seq model and loading embeddings from XLM-RoBERTa

Is it possible to feed embeddings from XLM- RoBERTa to transformer seq2seq model? I'm working on NMT that translates verbal language sentences to sign language sentences (e.g Input: He sells food. ...
4
votes
2answers
208 views

Why does vanilla transformer has fixed-length input?

I know that in the math on which the transformer is based there is no restriction on the length of input. But I still canā€™t understand why we should fix it in the frameworks (PyTorch). Because of this ...
0
votes
0answers
34 views

Seeking your advice on XLM-R for NMT

I want to use XLM-R for neural machine translation b/n the same low resource language? For example: Input-> He sells food(in Catalan) Output-> Food he sells(in Catalan) Anyone having code example/...
0
votes
1answer
262 views

How do Bahdanau - Luong Attentions use Query, Value, Key vectors?

In the latest TensorFlow 2.1, the tensorflow.keras.layers submodule contains AdditiveAttention() and ...
1
vote
1answer
21 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
2
votes
1answer
195 views

Why does the transformer positional encoding use both sine and cosine?

In the transformer architecture they use positional encoding (explained in this answer and I get how it is constructed. I am wondering why it needs to use both sine and cosine though instead of just ...
0
votes
2answers
75 views

How are Q, K, and V Vectors Trained in a Transformer Self-Attention?

I am new to transformers, so this may be a silly question, but I was reading about transformers and how they use attention, and it involves the usage of three special vectors. Most articles say that ...
1
vote
1answer
60 views

What is the feedforward network in a transformer trained on?

After reading the 'Attention is all you need' article, I understand the general architecture of a transformer. However, it is unclear to me how the feed forward neural network learns. What I learned ...
0
votes
0answers
25 views

Pretrained Models for Keyword-Based Text Generation

I'm looking for an implementation that allows me to generate text based on a pre-trained model (e.g. GPT-2). An example would be gpt-2-keyword-generation (click here for demo). As the author notes, ...
0
votes
0answers
104 views

Cross layer parameter sharing in ALBERT Model

I am reading the paper "ALBERT: LITE BERT FOR SELF-SUPERVISED LEARNING OF LANGUAGE REPRESENTATIONS". ALBERT uses cross layer parameter sharing to improve the model performance. I don't understand how ...
1
vote
0answers
66 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
1
vote
1answer
57 views

How do I implement Dual-encoder model in Pytorch?

I am trying to implement the paper titled Learning Cross-lingual Sentence Representations via a Multi-task Dual-Encoder Model. Here the encoder and decoder share the same weights but I am unable to ...
4
votes
2answers
91 views

Why Decision Tree Classifier is not working with categorical value?

I am learning my way through this, so please be easy on me if you find any mistakes, I could really use a professional opinion here. Thx. I am trying to model a Decision Tree Classifier as part of an ...
1
vote
1answer
24 views

Measuring quality of answers from QnA systems

I am having a question answering system which is using Seq2Seq kind of architecture. Actually it is a transformer architecture. When a question is asked it gives startposition and endposition of ...
3
votes
1answer
723 views

Proper masking in the transformer model

Concerning the transformer model, a mask is used to mask out attention scores (replace with 1e-9) prior to the matrix multiplication with the value tensor. Regarding the masking, I have 3 short ...
0
votes
1answer
14 views

the library 'transformers' works also with older version of Tensorflow?

i am working with Tensorflow version 1.14 and i would like to use the bert embedding. In order to do so, i was thinking to use the transformers library( https://pypi.org/project/transformers/) but i ...

1
2