11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 23.8k
6 votes
Accepted

Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
noe's user avatar
  • 23.8k
4 votes
Accepted

How to improve language model ex: BERT on unseen text in training?

In order to make your model more robust to different wordings, you may try with data augmentation techniques, that is, creating variations of your sentences and adding them to the training set with ...
noe's user avatar
  • 23.8k
3 votes

How to improve language model ex: BERT on unseen text in training?

It appears that your model is failing to generalize. One option is to increase the amount and quality of the training data. Other options include large-scale language model specific regularization ...
Brian Spiering's user avatar
2 votes

How does T5 model work on input and target data while transfer learning?

T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can ...
Jindřich's user avatar
  • 1,691
2 votes
Accepted

How to measure the accuracy of an NLP paraphrasing model?

Evaluation should always be specific to the target task and preferably rely on some unseen test set. The target task is paraphrasing, so the evaluation should be designed to check externally how good ...
Erwan's user avatar
  • 25k
2 votes
Accepted

How to get all 3 labels' sentiment from finbert instead of the most likely label's?

You can get the scores for all labels as follows: ...
Oxbowerce's user avatar
  • 7,157
2 votes
Accepted

What did Sentence-Bert return here?

There are two valid inputs to MPNet's tokenizer: Union[TextInputSequence, Tuple[InputSequence, InputSequence]] When you give a list of tuples as an input, from each tuple only the first two ...
Pushpam Punjabi's user avatar
2 votes

Possible NLP approaches to extract 'goals' from text

Well, A quick approach to this is using named entity recognition and POS tagging to identify key phrases in the text that are likely to be goals. For example, you might look for phrases that contain ...
Vic's user avatar
  • 196
2 votes

Extract the embedding from a specific layer of MarianModel

You are passing discrete tokens as input to an attention layer that expects vectors of real numbers as inputs. The error you are getting tells you that the layer does not expect an input parameter ...
noe's user avatar
  • 23.8k
2 votes
Accepted

Creating class labels for custom DataSets efficiently (HuggingFace)

This is a coding style issue, so people may well have different opinions! But I don't see any problem with the way you've coded it. If you really want to reduce the number of lines of code you could ...
Lynn's user avatar
  • 1,176
2 votes
Accepted

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

Resumes are quite different from classic text because there are many proper nouns (names, companies, places, etc.) and other data difficult to classify (phone numbers, marks, age, etc.). That's why ...
Nicolas Martin's user avatar
2 votes

BERT - How Question answering is different than classification

For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position). In the source ...
Astariul's user avatar
  • 1,004
2 votes

Dynamic batching and padding batches for NLP in deep learning libraries

That is commonly called sequence packing, creating a consistent-sized data structure composed of different, variable length sequences. Sequence packing has the potential to speed up training by ...
Brian Spiering's user avatar
2 votes

Open-Source Large Language Models (LLM): Your experience and recommendation

You should check the Open LLM Leaderboard from Huggingface. They maintain a ranking of open LLMs in different tasks. The ones finetuned to follow instructions are marked. When clicking on each model, ...
noe's user avatar
  • 23.8k
2 votes
Accepted

Can I run falcon-7b on a free google colab?

Yes there are a couple things you can do to fit the model into google colab's disk. Reduce the batch size for train and test. This will reduce the GPU memory used for each epoch. Default size set in ...
spectre's user avatar
  • 1,896
1 vote

Dynamic batching and padding batches for NLP in deep learning libraries

Great questions! When training a model with batches of data with padded inputs, the padded tokens will take up GPU memory. This is because the padding zeros are still being stored in memory, even if ...
Pluviophile's user avatar
  • 3,598
1 vote

error useing soft max gives outputs greater than 1

This piece of Python code is what you described: import torch a = torch.tensor([[-3.7550,-4.4172,7.8079]]) b = torch.softmax(a, 1) print(b) print(torch.sum(b)) It ...
noe's user avatar
  • 23.8k
1 vote

Do I need training data in multiple languages for a multilingual transformer?

As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the ...
noe's user avatar
  • 23.8k
1 vote

Transformer similarity fine-tuned way too often predicts pairs as similar

For NLP related tasks the transformer tries it's best to match your output distribution but as with all ml tasks it will fail on some parts of your data. Your task is somewhat similar to bert's next ...
Gary Ong's user avatar
1 vote

How to prepare texts to BERT/RoBERTa models?

You can use existing libraries to tokenize. From the docs on Github: For sentence-level tasks (or sentence-pair) tasks, tokenization is very simple. Just follow the example code in run_classifier.py ...
Peter's user avatar
  • 7,366
1 vote

Get sentence embeddings of transformer-based models

Usually, padding is excluded from mean pooling. One approach to derive sentence embeddings by mean pooling excluding padding tokens can be taken from Sentence Transformers. In their pooling source ...
Jonathan's user avatar
  • 5,360
1 vote

Question answering bot: EM>F1, does it make sense?

Metrics for Q&A F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer EM Score(exact match): which is the number of answers ...
Archana David's user avatar
1 vote

Question answering bot: EM>F1, does it make sense?

EM (exact match) and F1 scores are typically calculated on different levels. EM is calculated on the character level. F1 is calculated on individual word level. Almost always, EM will be lower than F1....
Brian Spiering's user avatar
1 vote

Masked Language Modeling on Domain-specific Data

First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. ...
Fatemeh Rahimi's user avatar
1 vote

dealing with HuggingFace's model's tokens

For your first question, you can check if the tokenizer covers a certain string with the following: ...
doguaraci's user avatar
1 vote

How to do NER predictions with Huggingface BERT transformer

Once the training is completed, use your trained model instance in a NER pipeline, using the same tokenizer as before: ...
Contestosis's user avatar
1 vote
Accepted

Which script can be used to finetune BERT for SQuAD question answering in Hugging Face library?

A recent PR changed the location of the scripts you are looking for to examples/legacy/question-answering
noe's user avatar
  • 23.8k

Only top scored, non community-wiki answers of a minimum length are eligible