Questions tagged [huggingface]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
6 views

Unable to debug where torch Adam optimiser is going wrong

I was implementing a training loop in vscode. I have created a Adam optimizer using XLM-Roberta model as follows: ...
user avatar
  • 137
0 votes
0 answers
5 views

Conversational model returns empty string after a while

I've been experinmenting with Huggingface models and I've set up a chatbot with DialoGPT. It works pretty well, but after a while it stops answering and just returns empty strings. Before this it will ...
user avatar
  • 1,248
1 vote
0 answers
11 views

How to train a Task Specific Knowledge Distillation model using Hugging face model

I was referring to this code: https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/knowledge-distillation.ipynb From @philschmid I could follow most of the ...
user avatar
  • 257
1 vote
0 answers
31 views

How to save hugging face fine tuned model using pytorch and distributed training

I am fine tuning masked language model from XLM Roberta large on google machine specs. When I copy the model using gsutil and subprocess from container to GCP ...
user avatar
  • 257
0 votes
0 answers
35 views

Hugging face Model Output 'last_hidden_state'

I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state. These are my questions What is the use of the ...
user avatar
  • 17
0 votes
0 answers
11 views

How to use label smoothing for single label classification in hugging face models

I am training a binary class classification model using Roberta-xlm large model. I am using training data with hard labels as either ...
user avatar
  • 257
0 votes
1 answer
18 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
user avatar
  • 101
0 votes
0 answers
20 views

Finetune XLM-RoBERTa on Tensorflow

I want to finetune pre-trained XLM-RoBERTa from HuggingFace for Text classification. I have categorical data in English. I want to finetune model on Tensorflow-keras. Can anyone let me know how can I ...
user avatar
1 vote
1 answer
64 views

How to prepare texts to BERT/RoBERTa models?

I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences. I want to train a language ...
user avatar
  • 141
0 votes
0 answers
119 views

Finetune XLM-RoBERTa on TF-keras for text classification

I am trying to finetune pre-trained XLM-RoBERTa on Tensorflow-keras. I am using dataset in English for text classification. I have used xlm-roberta-base tokenizer to tokenize the sentences. I am using ...
user avatar
0 votes
0 answers
235 views

HuggingFace transformer: CUDA out memory only when performing hyperparameter search

I am working with a GTX3070, which only has 8GB of GPU RAM. When I am running using trainer.train(), I run fine with a maximum batch size of 7 (6 if running in Jupiter notebook). However, when I ...
user avatar
0 votes
1 answer
23 views

Get sentence embeddings of transformer-based models

I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...). I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
user avatar
  • 145
0 votes
0 answers
15 views

If Bert can handle only 512 inputs. Why you can provide such long contexts in QA Pipeline?

For example, I use Pipeline from Huggingface Transformers to use a QA model card like this. ...
user avatar
2 votes
2 answers
94 views

How to improve language model ex: BERT on unseen text in training?

I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
user avatar
  • 123
0 votes
1 answer
221 views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
user avatar
  • 101
0 votes
0 answers
33 views

Code sanity check with one batch overfitting. Good value for loss?

In my current setup, I am trying to train a simple Bert model (DistilBert) for a classification task with 30 classes. As is common, I performing a quick sanity check whether my code actually properly ...
user avatar
1 vote
0 answers
79 views

How to Fine Tune a BERT model for sentiment analysis to get the best f1 score

I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score. More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
user avatar
  • 199
0 votes
0 answers
32 views

Chatbot NLP model for inference on 11 GB GPU?

I’ve just found the amazing Huggingface library. It is an awesome piece of work. I would like to train a chatbot on some existing dataset or several datasets (e.g. the Pile). For training (or fine-...
user avatar
  • 234
1 vote
1 answer
33 views

HuggingFace hate detection model

I am trying to train and evaluate a hate detection model using the HuggingFace Transformers library and this dataset. Model performance is secondary, just trying to get it going. I have preprocessed ...
user avatar
0 votes
1 answer
52 views

How does T5 model work on input and target data while transfer learning?

I am working on a project where I want the model to generate job description based on Role, Industry, Skills. I have trained my data and got the resultant output. I ...
user avatar
  • 33
0 votes
0 answers
472 views

Huggingface - TypeError: 'TensorSliceDataset' object is not subscriptable

I'm trying to make my own model for translate a language to another with T5ForConditionalGeneration and Huggingface using no pretrained model (I need to use my own dataset and tokenizer because no ...
user avatar
1 vote
0 answers
18 views

Can a reformer model really handle long-range dependency?

I read this article about new attention model called Reformer. Here is the main strength of this model: The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
user avatar
0 votes
0 answers
27 views

How to generate sentences based on words?

I have a dataframe which has columns Role Name, Technical Skills, Soft Skills and average experience. I have to use these words ...
user avatar
  • 33
0 votes
0 answers
16 views

Multi TextClassification using DistillBERT

I am building a text classifier using DistillBERT from Huggingface which classify input text into Geography, Environment and Science , How could I increase the accuracy for first two labels as both ...
user avatar
1 vote
0 answers
106 views

Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?

I'm doing sentiment analysis of Spanish tweets. After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
user avatar
  • 131
1 vote
1 answer
209 views

How to use is_split_into_words with Huggingface NER pipeline

I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train. My incoming text has already been split into words. When tokenizing during ...
user avatar
0 votes
0 answers
30 views

Which steps are involved in sentiment analysis with Huggingface Transformers?

I want to perform a sentiment analysis of a dataset of (Spanish) tweets about COVID-19 vaccines. I've already scraped the tweets and identified a pretrained model I can use for Spanish. What I don't ...
user avatar
  • 131
0 votes
0 answers
12 views

Optimal method(s) to monitor attention matrices when doing training/inference using BERT-type models from transformers

Our team is using BERT/Roberta from the huggingface transformers library for sequence-classification (amongst other tasks). We are looking for an efficient way to monitor the attention matrices so as ...
user avatar
  • 199
0 votes
0 answers
65 views

Training Loss increases, but Validation Loss decreases

I am finetuning a T5 transformer model on a sequence to sequence task. My program outputs the training and validation loss every 500 optimization steps. However, when I first started training the ...
user avatar
4 votes
1 answer
90 views

How to measure the accuracy of an NLP paraphrasing model?

I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
user avatar
0 votes
0 answers
6 views

How do pretrained models using SQUAD dataset work on an any other dataset?

I see in some Kaggle contests people have used models pretrained in SQUAD dataset for building QA systems for the dataset given in the contest. How does this work? How can a pretrained model in a ...
user avatar
1 vote
0 answers
21 views

Personal Project Classifying Bank Account Data - NLP Noob [closed]

Background: I'm looking to get up to speed with some of the newer ML classification techniques and keep my ML python fresh in my spare time/learn some new skills, so as a challenge I'm trying to see ...
user avatar
1 vote
1 answer
568 views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
user avatar
0 votes
0 answers
65 views

Are there any ways to check default values of pre-trained models before fine-tuning?

Background According to the instruction on Hugging Face page, I'm trying to fine tune pre-trained model for named entity recognition. I think I should try Transfer Learning for the first, but there is ...
user avatar
  • 129
0 votes
2 answers
102 views

Question answering bot: EM>F1, does it make sense?

I am fine-tuning a Question Answering bot starting from a pre-trained model from HuggingFace repo. The dataset I am using for the fine-tuning has a lot of empty answers. So, after the fine tuning, ...
user avatar
0 votes
0 answers
126 views

Pytorch DataLoader returns iterable , how and when to convert into a Tensor for Model Training

I have coded up two DataSet classes (one map-style and one iterator-style) to be used with DataLoader (either one is okay, but I just wanted to experiment and learn myself) on TEXT data. The data + ...
user avatar
  • 101
1 vote
0 answers
105 views

How to do batch inference on Hugging face pretrained models?

I want to do batch inference on MarianMT model. Here's the code: ...
user avatar
0 votes
0 answers
68 views

Does NSP task corrupt context during pre-training?

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
user avatar
  • 101
1 vote
0 answers
18 views

Train and validation sets splits using load_data

I'm using the package "datasets". The code I have: ...
user avatar
1 vote
0 answers
13 views

Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling

I intend to use 2 tasks of modelling including (a) Causal language modelling & (b) Mask language modelling for training my new added tokens My pseudo-code is below ...
user avatar
1 vote
1 answer
112 views

Masked Language Modeling on Domain-specific Data

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
user avatar
4 votes
1 answer
1k views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
user avatar
1 vote
1 answer
148 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
user avatar
1 vote
1 answer
1k views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
user avatar
1 vote
0 answers
39 views

Top-K vs AUC - communicating results and next steps [closed]

I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of ...
user avatar
  • 111
2 votes
1 answer
521 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
user avatar
  • 121
0 votes
0 answers
49 views

List of Google T5 possible operations

I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
user avatar
0 votes
1 answer
101 views

which script can be used to finetune BERT fro squad question answering in hugging face library

I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used ...
user avatar
0 votes
0 answers
112 views

BERT for classification model degenerates into all-positive predictions

As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
user avatar
1 vote
0 answers
143 views

Huggingface Library - Multi-document summarization

Can BART, PEGASUS ... etc. API in huggingface library be used to directly perform multi document summarization? (e.g. here: https://huggingface.co/transformers/model_doc/bart.html)
user avatar