Skip to main content
Share Your Experience: Take the 2024 Developer Survey
11 votes
Accepted

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

After a Googling around, I think this tutorial may suit your needs. However, it seems you have a misconception about the Transformer decoder: in training mode there is no iteration at all. While LSTM-...
noe's user avatar
  • 26.9k
6 votes
Accepted

Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
noe's user avatar
  • 26.9k
5 votes
Accepted

Should you care about truncation and padding in an LLM even if it has a very large tokenizer.max_length so that truncation will never happen?

The large number you are seeing is not the maximum length, but the maximum representable integer at that precision. It's there because no maximum length has been set. The original GPT-2 has a maximum ...
noe's user avatar
  • 26.9k
4 votes
Accepted

How to improve language model ex: BERT on unseen text in training?

In order to make your model more robust to different wordings, you may try with data augmentation techniques, that is, creating variations of your sentences and adding them to the training set with ...
noe's user avatar
  • 26.9k
3 votes

How to improve language model ex: BERT on unseen text in training?

It appears that your model is failing to generalize. One option is to increase the amount and quality of the training data. Other options include large-scale language model specific regularization ...
Brian Spiering's user avatar
3 votes
Accepted

Since LoRA parameters are randomly initialized, shouldn't that mean that initially breaks a models output?

Not all the LoRA parameters are initialized randomly, only one of the matrices of the decomposition is. From the original LoRA article: We use a random Gaussian initialization for A and zero for B, ...
noe's user avatar
  • 26.9k
2 votes

How does T5 model work on input and target data while transfer learning?

T5 is in fact a sequence-to-sequence model, it has an encoder that generates some hidden states representing the input and a decoder that generates the output. When you fine-tune the model you can ...
Jindřich's user avatar
  • 1,751
2 votes
Accepted

How to use is_split_into_words with Huggingface NER pipeline

evalute.evaluator.token_classification.py has this line: data = data.map(lambda x: {input_column: join_by.join(x[input_column])}) So even though the ...
Trail Map's user avatar
2 votes
Accepted

How to measure the accuracy of an NLP paraphrasing model?

Evaluation should always be specific to the target task and preferably rely on some unseen test set. The target task is paraphrasing, so the evaluation should be designed to check externally how good ...
Erwan's user avatar
  • 25.5k
2 votes

BERT - How Question answering is different than classification

For Question Answering, you need 2 logits : one for the start position, one for the end position. Based on these 2 logits, you have an answer span (denoted by the start/end position). In the source ...
Astariul's user avatar
  • 1,004
2 votes
Accepted

How to get all 3 labels' sentiment from finbert instead of the most likely label's?

You can get the scores for all labels as follows: ...
Oxbowerce's user avatar
  • 7,582
2 votes
Accepted

What did Sentence-Bert return here?

There are two valid inputs to MPNet's tokenizer: Union[TextInputSequence, Tuple[InputSequence, InputSequence]] When you give a list of tuples as an input, from each tuple only the first two ...
Pushpam Punjabi's user avatar
2 votes

Possible NLP approaches to extract 'goals' from text

Well, A quick approach to this is using named entity recognition and POS tagging to identify key phrases in the text that are likely to be goals. For example, you might look for phrases that contain ...
Vic's user avatar
  • 306
2 votes

Extract the embedding from a specific layer of MarianModel

You are passing discrete tokens as input to an attention layer that expects vectors of real numbers as inputs. The error you are getting tells you that the layer does not expect an input parameter ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Creating class labels for custom DataSets efficiently (HuggingFace)

This is a coding style issue, so people may well have different opinions! But I don't see any problem with the way you've coded it. If you really want to reduce the number of lines of code you could ...
Lynn's user avatar
  • 1,307
2 votes
Accepted

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

Resumes are quite different from classic text because there are many proper nouns (names, companies, places, etc.) and other data difficult to classify (phone numbers, marks, age, etc.). That's why ...
Nicolas Martin's user avatar
2 votes

Dynamic batching and padding batches for NLP in deep learning libraries

That is commonly called sequence packing, creating a consistent-sized data structure composed of different, variable length sequences. Sequence packing has the potential to speed up training by ...
Brian Spiering's user avatar
2 votes
Accepted

Can I run falcon-7b on a free google colab?

Yes there are a couple things you can do to fit the model into google colab's disk. Reduce the batch size for train and test. This will reduce the GPU memory used for each epoch. Default size set in ...
spectre's user avatar
  • 2,105
2 votes

Open-Source Large Language Models (LLM): Your experience and recommendation

You should check the Open LLM Leaderboard from Huggingface. They maintain a ranking of open LLMs in different tasks. The ones finetuned to follow instructions are marked. When clicking on each model, ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Understanding processors in huggingface tokenizer library

The answer is in your screenshot: For each of these (special tokens and sentences), we also specify the corresponding token type ID after a colon Therefore, the :0...
noe's user avatar
  • 26.9k
2 votes

How can you get a Huggingface fine-tuning model with the Trainer class from your own text where you can set the arguments for truncation and padding?

If you use transformers example scripts, for example, summarization, you can control padding and truncation with command line arguments --pad_to_max_length, ...
Valentas's user avatar
  • 1,209
2 votes
Accepted

Falcon-7B llm giving random output

You are using the GPT-2 tokenizer instead of the model's own tokenizer. Try getting the tokenizer from the model instead: ...
noe's user avatar
  • 26.9k
2 votes
Accepted

Getting rid of the warning "The following columns ... have been ignored" and "ValueError: batch_size should be a positive integer value ..."

I had an outdated version of transformers. I upgraded thetransformers package from: ...
questionto42's user avatar
2 votes
Accepted

Generate VTT file from speech to text

Have you looked at using the return_timestamps keyword? It seems you can also get the timestamps on either the word on text segment level (for the Whisper model at ...
Oxbowerce's user avatar
  • 7,582
2 votes

Public Email Classification Dataset but not Spam vs Ham

If I am understanding correctly, you want to create a model that takes an email body and assigns some probability to a pre-specified set of classes (feedback, complaint, lost and found, etc.) ...
AndrewJaeyoung's user avatar
2 votes
Accepted

How do I get model.generate() to omit the input sequence from the generation?

Given that Llama is a decoder-only model, the output of the model (i.e. outputs) is just the whole sequence. You can, nevertheless, decode only the new tokens ...
noe's user avatar
  • 26.9k
1 vote

error useing soft max gives outputs greater than 1

This piece of Python code is what you described: import torch a = torch.tensor([[-3.7550,-4.4172,7.8079]]) b = torch.softmax(a, 1) print(b) print(torch.sum(b)) It ...
noe's user avatar
  • 26.9k
1 vote

How can someone evaluate llama index model?

Is there any other way to evaluate model ? Assuming you have a test set with gold answers, you can use a text generation metric. See Evaluation of Text Generation: A Survey. Typical metrics: TF-IDF ...
Franck Dernoncourt's user avatar
1 vote
Accepted

LMM Fine Tuning - Supervised Fine Tuning Trainer (SFTTrainer) vs transformers Trainer

The short answer is that a Supervised Fine Tuning Trainer (SFTTrainer) is used for Instruct Fine Tuning. The HuggingFace library SFTTrainer has also support for training with QLoRA (4-bit Quantised ...
Alex Punnen's user avatar
1 vote

Fine-tuning a pre-trained LLM for question-answering

Check the steps at the Huggingface beginner's guide Question Answering with SQuAD 2.0 to begin with a normal question answering model. Have some look at the Fine-tuning guide at OpenAI as well (which ...
questionto42's user avatar

Only top scored, non community-wiki answers of a minimum length are eligible