Jindřich
  • Member for 2 years, 7 months
  • Last seen this week
Overfitting with text classification using Transformers
8 votes

Your model is overfitting. You should try standard methods people use to prevent overfitting: Larger dropout (up to 0.5), in low-resource setups word dropout (i.e., randomly masking input tokens) ...

View answer
Does BERT use GLoVE?
Accepted answer
5 votes

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...

View answer
Transformer masking during training or inference?
Accepted answer
3 votes

The trick is that you do not need masking at inference time. The purpose of masking is that you prevent the decoder state from attending to positions that correspond to tokens "in the future"...

View answer
Transformer seq2seq model and loading embeddings from XLM-RoBERTa
3 votes

It is indeed possible, but the question is if it is a good idea. FairSeq already contains a pre-trained XLM-R model, you can use by creating a new model: just copy the most suitable existing one and ...

View answer
How are Q, K, and V Vectors Trained in a Transformer Self-Attention?
3 votes

These matrices are not learned parameters but are a result of previous (yet parameterized) computations. In self-attentive layers, are all three of them the same, they are the outputs of the previous ...

View answer
BERT vs GPT architectural, conceptual and implemetational differences
2 votes

To start with your last question: you correctly say that BERT is an encoder-only model trained with the masked language-modeling objective and operates non-autoregressively. GPT-2 is a decode-only ...

View answer
Does BERT need supervised data only when fine-tuning?
Accepted answer
2 votes

The distinction between supervised and unsupervised is a little bit tricky here. BERT pre-training is unsupervised with respect to the downstream tasks, but the pre-training itself is technically a ...

View answer
Smart sentence segmentation not splitting on abbreviations
2 votes

Neural tools trained on Universal Dependencies corpora use learned models for tokenization and sentence-spliting. Two I know of are: UDPipe – developed at Charles University in Prague. Gets very good ...

View answer
Transformer-based architectures for regression tasks
2 votes

In the simplest case, doing regression with Transformers is just a matter of changing the loss function. BERT-like models that use the representation of the first technical token as an input to the ...

View answer
SpaCy string store
2 votes

Spacy uses a hash function that assigns an integer to any Unicode string, it is not an index in vocabulary it just a random integer that is used internally for better efficiency. It is a hash function,...

View answer
Attention mechanism in Tensorflow 2
Accepted answer
2 votes

In self-attention, it is not the decoder attending the encoder, but the layer attends itself, i.e., the queries and values are the same. In practice, this is usually done in the multi-head setup. You ...

View answer
What is the advantage of positional encoding over one hot encoding in a transformer model?
2 votes

The theoretical advantage should be that the network should be able to grasp the pattern from the encoding and thus generalize better for longer sentences. With one-hot position encoding, you would ...

View answer
When to use GloVe vocabulary vs. building a vocabulary from the training data?
Accepted answer
1 votes

The arguments that you say are pretty much correct. The primary reason for using pre-trained embeddings is typically the lack of task-specific training data. In tasks that (at least for some languages)...

View answer
Optimal batch size and number of epoch for BERT
Accepted answer
1 votes

The general answer is the same as with everything in machine learning: it depends on the particular task. With Transformers, people tend to recommend larger batch sizes, typically thousands of tokens ...

View answer
How does T5 model work on input and target data while transfer learning?
1 votes

T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can ...

View answer
why is the BERT NSP task useful for sentence classification tasks?
Accepted answer
1 votes

The motivation is that the [CLS] embedding should contain "a summary" of both sentences to be able to decide if they follow each other or not. However, in follow-up papers such as RoBERTa or ...

View answer
For text classification, would a BoW or Word Embeddings based model ever be better than a Language Model?
1 votes

Unfortunately, there is little theoretical knowledge about what complex neural networks do. Transformers are known to be universal approximations, so in theory they can learn to do any function with ...

View answer
How to capture the detail of an attribute in a sentence?
1 votes

You can try using dependency parsing for that, e.g., as implemented in Spacy or any other NLP toolkit. The details should be adjective modifiers (dependency label amod) of what you call attribute that ...

View answer
How to cluster words automatically?
1 votes

You can get a dendrogram from any hierarchical clustering method. The tricky thing here is how to compute the distances between the words. If efficiency is your main concern, I would consider using ...

View answer
How to interpret the value of categorical cross entropy?
1 votes

There is usually no straightforward interpretation of what cross-entropy means in the context of the given task. In practice, it is more important to follow the trend of how the cross-entropy develops ...

View answer
Difference between zero-padding and character-padding in Recurrent Neural Networks
Accepted answer
1 votes

If implemented properly, there should be no difference. The very first thing that happens with the indices is corresponding embeddings are loaded. From this perspective, there is no difference between ...

View answer
Why are mBART50 language codes in an unusual format?
Accepted answer
1 votes

It encodes the language and its regional variant, the same way as locales are encoded. hi_IN then means Hindi as spoken in India, en_US would mean American English, en_GB British English. My guess is ...

View answer
Cross between an edit distance algorithm and a phonetic algorithm
Accepted answer
1 votes

The operations in the edit distance can have different weights that you can try to set manually. In Python, you can use e.g., strsympy or weighted-levenshtein package. There is also Learnable Edit ...

View answer
Pytorch Luong global attention: what is the shape of the alignment vector supposed to be?
Accepted answer
1 votes

You answered yourself [sequence length, 1] is correct assuming you work with a single sentence. (Or actually, the 1 dimension depends on implementation.) In practice, the data is typically batched, so ...

View answer
Context Based Embeddings vs character based embeddings vs word based embeddings
Accepted answer
1 votes

Context-based or contextual means that the vector contains information about the use of the word in a context of a sentence (or rarely a document). It thus does not make sense to talk about the word ...

View answer
BERT minimal batch size
Accepted answer
1 votes

Small mini-batch size leads to a big variance in the gradients. In theory, with a sufficiently small learning rate, you can learn anything even with very small batches. In practice, Transformers are ...

View answer
use the same gradient to maximize one part of the model and minimize another part of the same model
Accepted answer
1 votes

The trick you are looking for is called the Gradient Reversal Layer. It is a layer that does nothing (i.e., identity) in the forward pass, but it reverts the sign of the gradient, so everything behind ...

View answer
What would be the target input for Transformer Decoder during test phase?
Accepted answer
1 votes

At training time, the input to the decoder is the target sentence tokens, which are indeed unknown at the test time. What you call the second input are the desired outputs, which are not usually ...

View answer
Getting sentence embeddings with sentence_transformers
1 votes

The encode method only works with single sentences as strings, i.e., you need to call it for each sentence independently: embeddings = [model.encode(s) for s in sentences]

View answer
Splitting into multiple heads -- multihead self attention
1 votes

In principle, the pseudocode is correct, but it is not how it is implemented. The projection and dot-product attention can be done efficiently using matrix multiplication only for all heads ...

View answer