Questions tagged [transformer]

Use for questions related to the Transformer (based on encoder-decoder) architecture in machine learning.

Filter by
Sorted by
Tagged with
0
votes
0answers
22 views

What would be the target input for Transformer Decoder during test phase?

The Transformer Decoder takes in two inputs, the encoder's output, and the target sequence. How the target is fed into the decoder has been provided in this answer I am having confusion about what ...
4
votes
1answer
96 views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
0
votes
1answer
20 views

How to treat data transformation choices as hyperparemeters?

While reading the book hands-on ML by Aurelien Geron, I came across this line- Treat your data transformation choices as hyperparameters, especially when you are not sure about them (e.g., if you’re ...
0
votes
0answers
19 views

Bert question answering start_positions is larger than end_positions [closed]

I want to fine-tune Bert on question answering for a closed domain, so I started by discovering how it works first, i executed the code bellow but the result is no write, the start position is larger ...
1
vote
0answers
25 views

Question about BERT embeddings with high cosine similarity

Under what circumstances would BERT assign two occurrences of the same word similar embeddings? If those occurrences are contained within similar syntactic relations with their co-occurrents?
1
vote
1answer
26 views

Does finetuning BERT involving updating all of the parameters or just the final classification layer?

Currently learning and reading about transformer models, I get that during the pretraining stage the BERT model is trained on a large corpus via MLM and NSP. But during finetuning, for example trying ...
0
votes
0answers
10 views

Reformer the Efficient Transformer and Image Size Limits

I'm currently trying to use the Reformer: Image Generator with my own dataset. The colab notebook for the model is here: https://colab.research.google.com/github/google/trax/blob/master/trax/models/...
2
votes
0answers
46 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
2
votes
1answer
22 views

Transformer masking during training or inference?

I'm working through Attention is All you Need, and I have a question about masking in the decoder. It's stated that masking is used to ensure the model doesn't attend to any tokens in the future (not ...
0
votes
1answer
27 views

Why does the non autoregresive transfomer model in fairseq require the prev_output_tokens input?

fairseq includes an implementation of a non autoregressive transformer - which (as much as I understand) means that the whole output sequence is generated in a single forward run (in contrast to ...
0
votes
1answer
21 views

Splitting into multiple heads — multihead self attention

So, I have a doubt in Attention is all you need: The implementation of transformers on tensorflow's official documentation says: Each multi-head attention block gets three inputs; Q (query), K (key), ...
0
votes
2answers
35 views

What are the hidden states in the Transformer-XL? Also, how does the recurrence wiring look like?

After exhaustively reading the many blogs and papers on Transformers-XL, I still have some questions before I can say that I understand Transformer-XL (and by extension XLNet). Any help in this regard ...
1
vote
1answer
30 views

Based on transformer, how to improve the text generation results?

If I do not pretrain the text generation model like BART, how to improve the result based on transformer like tensor2tensor? What are the improvement ideas for transformer in text generation task?
0
votes
1answer
74 views

Overfitting while fine-tuning pre-trained transformer

Pretrained transformers (GPT2, Bert, XLNET) is popular and useful because of their transfer learning capabilities. Just to remind: The goal of Transfer learning is is to transfer knowledge gained from ...
2
votes
1answer
29 views

Explanation about i//2 in positional encoding in tensorflow tutorial about transformers

I was implementing the transformer architecture in tensorflow. I was following the tutorial : https://www.tensorflow.org/tutorials/text/transformer#setup_input_pipeline They implement the positional ...
0
votes
1answer
25 views

NLP Transformers - understanding the multi-headed attention visualization (Attention is all you need)

I am new to NLP and I just finished reading the paper "Attention is all you need". I'm struggling to understand the interpretability of the multi-headed attention, and specifically how these ...
-1
votes
1answer
98 views

For NLP, is GPT-3 better than RoBERTa? [closed]

I am learning deep learning and I want to get into NLP. I have done LSTM, and now I am learning about vectorisation and transformers. Can you please tell me, which algorithm is more effective and ...
0
votes
1answer
11 views

Question of pretraining text-generation task, it seems that pretraining is not work for a small model?

My task is to generate keywords from sentences. I pretrain a text-generation model. I mask the sentences' tokens and predict the whole sentences' tokens. Pretraining batch_size = 8 and step = 1000000 ...
0
votes
0answers
29 views

huggingface bert - need help on working of code

Looking for some explanation of understanding of the BERT implementation by huggingface. I would explain my understanding below and then ask question: Below is code for question answering ...
0
votes
0answers
11 views

Question Answering - Was there any direct model before BERT?

This is a follow up question after question-answering-without-bert-transformers . While in search of a question answering mechanism without transformers, I am hitting dead ends. Old question answering ...
0
votes
0answers
44 views

Hierarchical transformer for document classification: error with model implementation, and advice on extracting attention weights

I am trying to implement a hierarchical transformer for document classification in Keras/tensorflow, in which: (1) a word-level transformer produces a representation of each sentence, and attention ...
0
votes
1answer
29 views

BERT reasoning capabilities

I'm working on a Twitter classification task and while analyzing the errors I found quite a few strange predictions. I'm searching for a tool (preferably open-source) similar to https://...
1
vote
1answer
19 views

what is the difference between positional vector and attention vector used in transformer model?

what is the difference between positional vector and attention vector used in transformer model ? , i saw a video in youtue and the defintion for positional vector was give as :* "vector that ...
0
votes
1answer
88 views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
1
vote
2answers
107 views

Next sentence prediction in RoBERTa

I'm trying to wrap my head around the way next sentence prediction works in RoBERTa. Based on their paper, in section 4.2, I understand that in the original BERT they used a pair of text segments ...
0
votes
0answers
40 views

What is the difference between register_buffer() and parameter.detach() in PyTorch?

I am writing a PositionalEmbedding() module which is an implementation based on "Attention Is All You Need" using PyTorch. According to the paper, there ...
0
votes
0answers
16 views

Calculating key and value vector in the Transformer's decoder block

I am implementing the transformer model in Pytorch by following Jay Alammar's post and the implementation here. My question is regarding the input to the decoder layer. As shown in the diagram above, ...
0
votes
0answers
18 views

What are the simplest methods for the label noise problem?

If I have enough low quality data from unsupervised methods or rule-based methods. I read from https://github.com/subeeshvasu/Awesome-Learning-with-Label-Noise ,but these methods are a little complex ...
0
votes
1answer
40 views

How to understand Inconsistent and ambiguous dimensions of matrices used in the Attention layer?

Attention-scoring mechanism seems to be a commonly-used component in various seq2seq models, and I was reading about the original "Location-based Attention" in Bahadanau well-known paper at https://...
2
votes
0answers
191 views

BPE vs WordPiece Tokenization - when to use / which?

What's the general tradeoff between choosing BPE vs WordPiece Tokenization? When is one preferable to the other? Are there any differences in model performance between the two? I'm looking for a ...
0
votes
1answer
31 views

Data quantity is not low but data quality is low, what are the best practices now?

Text classification task, if data quantity is low but data quality is not low. We could use data augment methods for improvement. But the situation is that data quantity is not low and data quality ...
1
vote
1answer
37 views

TensorFlow1.15, multi-GPU-1-machine, how to set batch_size?

The input function code: ...
0
votes
0answers
13 views

Why does the BERT NSP head linear layer have two outputs?

Here's the code in question. https://github.com/huggingface/transformers/blob/master/src/transformers/modeling_bert.py#L491 ...
2
votes
1answer
59 views

German Chatbot or conversational AI

I want to build a chatbot mostly BERT(Transformer) based in the German Language. But I do not find any German chatbot data set! So does it make sense to use google translator API to translate the ...
2
votes
1answer
157 views

Transformer-based architectures for regression tasks

As far as I've seen, transformer-based architectures are always trained with classification tasks (one-hot text tokens for example). Are you aware of any architectures using attention and solving ...
2
votes
1answer
37 views

Transformer decoder output - how is it linear?

I'm not quite sure how's the decoder output is flattened into a single vector. As from my understanding, if we input the encoder with a length N sentence, it's output is N x units (e.g. N x 1000), and ...
0
votes
1answer
37 views

sklearn ColumnTransformer creates new columns in output when there are overlapping columns between steps

I need to process some dataframe columns in different steps using ColumnTransformer. The first step process the date columns (timestamp) imputing missing values and the second step applies scaling to ...
3
votes
1answer
284 views

Is BERT a language model?

Is BERT a language model in the sense of a function that gets a sentence and returns a probability? I know its main usage is sentence embedding, but can it also provide this functionality?
1
vote
2answers
151 views

Does BERT use GLoVE?

From all the docs I read, people push this way and that way on how BERT uses or generates embedding. I GET that there is a key and a query and a value and those are all generated. What I don't know ...
0
votes
0answers
103 views

BERT Implementaion for Sequence Classification

I am trying to implement BERT using HuggingFace - transformers implementation. I am following two links: by analytics-vidhya and by HuggingFace Below is the code: ...
0
votes
0answers
52 views

HuggingFace/Transformers Implementation for Classification

I am trying to implement BERT using HuggingFace - transformers implementation. I am following two links: by analytics-vidhya and by HuggingFace If we consider inputs for both the implementations: 1) ...
0
votes
0answers
74 views

Positional Encoding of Categorical Features in a Time Series Transformer

I am training a Transformer for Multivariate Time Series prediction. I am working with Categorical features and I am thinking of using Positional Encoding to encode them instead of Embedding. Has ...
0
votes
1answer
484 views

Overfitting with text classification using Transformers

I am trying to make a binary text classification model by using the encoder part of the transformer and then using its output to feed into an LSTM network. However, I am not able to achieve good ...
1
vote
1answer
86 views

Can BERT be used for predicting words?

I have a question regarding the pre-training section (in particular, the Masked Language Model). In the example Let's stick to improvisation in this skit, by masking the word improvisation, after ...
1
vote
2answers
34 views

What are good toy problems for testing Transformer architectures?

I am testing various variants for Transformers and Transformer architectures. But training on full language tasks is a rather time consuming affair. What are good toy problems to test if a transformer ...
2
votes
1answer
47 views

Custom functions and pipelines

I'm not really used to working with pipelines, so I'm wondering how can I use custom functions and pipelines. Situation: I want to fill some missing values with the mean but using groups based on ...
0
votes
0answers
86 views

What is “position” in CNN (im2latex) for Positional Encoding?

I'm trying to build a model that maps images of math formulas into LaTeX markup. I found an acticle (https://arxiv.org/ftp/arxiv/papers/1908/1908.11415.pdf) that proposes an encoder-decoder ...
1
vote
2answers
35 views

In “Attention Is All You Need”, why are the FFNs in (2) the same as two convolutions with kernel size 1?

In addition, why do we need a FFN in each layer when we already have attention? Here's a screenshot of the relevant section from Vaswani et al. (2017):
0
votes
0answers
25 views

Transformers and BERT: dealing with possessives and apostrophes when encode

Let's consider two sentences: "why isn't Alex's text tokenizing? The house on the left is the Smiths' house" Now let's tokenize and decode: ...
0
votes
0answers
120 views

How to detokenize a BertTokenizer output?

For example, let's tokenize a sentece "why isn't Alex' text tokenizing": ...