11
votes
Accepted
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
After a Googling around, I think this tutorial may suit your needs.
However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
- 23.8k
6
votes
Accepted
Bert for QuestionAnswering input exceeds 512
The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
- 23.8k
4
votes
Accepted
How to improve language model ex: BERT on unseen text in training?
In order to make your model more robust to different wordings, you may try with data augmentation techniques, that is, creating variations of your sentences and adding them to the training set with ...
- 23.8k
3
votes
How to improve language model ex: BERT on unseen text in training?
It appears that your model is failing to generalize.
One option is to increase the amount and quality of the training data.
Other options include large-scale language model specific regularization ...
- 20.5k
2
votes
How does T5 model work on input and target data while transfer learning?
T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can ...
- 1,691
2
votes
Accepted
How to measure the accuracy of an NLP paraphrasing model?
Evaluation should always be specific to the target task and preferably rely on some unseen test set.
The target task is paraphrasing, so the evaluation should be designed to check externally how good ...
- 25k
2
votes
Accepted
How to get all 3 labels' sentiment from finbert instead of the most likely label's?
You can get the scores for all labels as follows:
...
- 7,157
2
votes
Accepted
What did Sentence-Bert return here?
There are two valid inputs to MPNet's tokenizer:
Union[TextInputSequence, Tuple[InputSequence, InputSequence]]
When you give a list of tuples as an input, from each tuple only the first two ...
2
votes
Possible NLP approaches to extract 'goals' from text
Well, A quick approach to this is using named entity recognition and POS tagging to identify key phrases in the text that are likely to be goals.
For example, you might look for phrases that contain ...
- 196
2
votes
Extract the embedding from a specific layer of MarianModel
You are passing discrete tokens as input to an attention layer that expects vectors of real numbers as inputs. The error you are getting tells you that the layer does not expect an input parameter ...
- 23.8k
2
votes
Accepted
Creating class labels for custom DataSets efficiently (HuggingFace)
This is a coding style issue, so people may well have different opinions! But I don't see any problem with the way you've coded it.
If you really want to reduce the number of lines of code you could ...
- 1,176
2
votes
Accepted
What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?
Resumes are quite different from classic text because there are many proper nouns (names, companies, places, etc.) and other data difficult to classify (phone numbers, marks, age, etc.).
That's why ...
- 4,589
2
votes
BERT - How Question answering is different than classification
For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position).
In the source ...
- 1,004
2
votes
Dynamic batching and padding batches for NLP in deep learning libraries
That is commonly called sequence packing, creating a consistent-sized data structure composed of different, variable length sequences. Sequence packing has the potential to speed up training by ...
- 20.5k
2
votes
Open-Source Large Language Models (LLM): Your experience and recommendation
You should check the Open LLM Leaderboard from Huggingface.
They maintain a ranking of open LLMs in different tasks. The ones finetuned to follow instructions are marked. When clicking on each model, ...
- 23.8k
2
votes
Accepted
Can I run falcon-7b on a free google colab?
Yes there are a couple things you can do to fit the model into google colab's disk.
Reduce the batch size for train and test. This will reduce the GPU memory used for each epoch. Default size set in ...
- 1,896
1
vote
Dynamic batching and padding batches for NLP in deep learning libraries
Great questions!
When training a model with batches of data with padded inputs, the padded tokens will take up GPU memory. This is because the padding zeros are still being stored in memory, even if ...
- 3,598
1
vote
error useing soft max gives outputs greater than 1
This piece of Python code is what you described:
import torch
a = torch.tensor([[-3.7550,-4.4172,7.8079]])
b = torch.softmax(a, 1)
print(b)
print(torch.sum(b))
It ...
- 23.8k
1
vote
Do I need training data in multiple languages for a multilingual transformer?
As far as I know, few multilingual models study their representation space to see if the representations of different languages occupy overlapping regions. The ones that do, usually find that the ...
- 23.8k
1
vote
Transformer similarity fine-tuned way too often predicts pairs as similar
For NLP related tasks the transformer tries it's best to match your output distribution but as with all ml tasks it will fail on some parts of your data. Your task is somewhat similar to bert's next ...
- 73
1
vote
How to prepare texts to BERT/RoBERTa models?
You can use existing libraries to tokenize.
From the docs on Github:
For sentence-level tasks (or sentence-pair) tasks, tokenization is
very simple. Just follow the example code in run_classifier.py ...
- 7,366
1
vote
Get sentence embeddings of transformer-based models
Usually, padding is excluded from mean pooling.
One approach to derive sentence embeddings by mean pooling excluding padding tokens can be taken from Sentence Transformers. In their pooling source ...
- 5,360
1
vote
Question answering bot: EM>F1, does it make sense?
Metrics for Q&A
F1 score: Captures the precision and recall that words chosen as being part of the answer are actually part of the answer
EM Score(exact match): which is the number of answers ...
- 1,229
1
vote
Question answering bot: EM>F1, does it make sense?
EM (exact match) and F1 scores are typically calculated on different levels. EM is calculated on the character level. F1 is calculated on individual word level.
Almost always, EM will be lower than F1....
- 20.5k
1
vote
Masked Language Modeling on Domain-specific Data
First I suggest reading the transformers paper. Couple of quick notes is that this model consists of an encoder and a decoder, and the original task the paper is trained on is machine translation. ...
- 549
1
vote
dealing with HuggingFace's model's tokens
For your first question, you can check if the tokenizer covers a certain string with the following:
...
- 11
1
vote
How to do NER predictions with Huggingface BERT transformer
Once the training is completed, use your trained model instance in a NER pipeline, using the same tokenizer as before:
...
- 181
1
vote
Accepted
Which script can be used to finetune BERT for SQuAD question answering in Hugging Face library?
A recent PR changed the location of the scripts you are looking for to examples/legacy/question-answering
- 23.8k
Only top scored, non community-wiki answers of a minimum length are eligible