Questions tagged [huggingface]
The huggingface tag has no usage guidance.
55
questions
0
votes
0
answers
6
views
Unable to debug where torch Adam optimiser is going wrong
I was implementing a training loop in vscode. I have created a Adam optimizer using XLM-Roberta model as follows:
...
- 137
0
votes
0
answers
5
views
Conversational model returns empty string after a while
I've been experinmenting with Huggingface models and I've set up a chatbot with DialoGPT. It works pretty well, but after a while it stops answering and just returns empty strings.
Before this it will ...
- 1,248
1
vote
0
answers
11
views
How to train a Task Specific Knowledge Distillation model using Hugging face model
I was referring to this code:
https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/knowledge-distillation.ipynb
From @philschmid
I could follow most of the ...
- 257
1
vote
0
answers
31
views
How to save hugging face fine tuned model using pytorch and distributed training
I am fine tuning masked language model from XLM Roberta large on google machine specs.
When I copy the model using gsutil and subprocess from container to GCP ...
- 257
0
votes
0
answers
35
views
Hugging face Model Output 'last_hidden_state'
I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state.
These are my questions
What is the use of the ...
- 17
0
votes
0
answers
11
views
How to use label smoothing for single label classification in hugging face models
I am training a binary class classification model using Roberta-xlm large model.
I am using training data with hard labels as either ...
- 257
0
votes
1
answer
18
views
Transformer similarity fine-tuned way too often predicts pairs as similar
I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data:
...
- 101
0
votes
0
answers
20
views
Finetune XLM-RoBERTa on Tensorflow
I want to finetune pre-trained XLM-RoBERTa from HuggingFace for Text classification. I have categorical data in English. I want to finetune model on Tensorflow-keras. Can anyone let me know how can I ...
1
vote
1
answer
64
views
How to prepare texts to BERT/RoBERTa models?
I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences.
I want to train a language ...
- 141
0
votes
0
answers
119
views
Finetune XLM-RoBERTa on TF-keras for text classification
I am trying to finetune pre-trained XLM-RoBERTa on Tensorflow-keras. I am using dataset in English for text classification. I have used xlm-roberta-base tokenizer to tokenize the sentences. I am using ...
0
votes
0
answers
235
views
HuggingFace transformer: CUDA out memory only when performing hyperparameter search
I am working with a GTX3070, which only has 8GB of GPU RAM. When I am running using trainer.train(), I run fine with a maximum batch size of 7 (6 if running in Jupiter notebook). However, when I ...
0
votes
1
answer
23
views
Get sentence embeddings of transformer-based models
I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...).
I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
- 145
0
votes
0
answers
15
views
If Bert can handle only 512 inputs. Why you can provide such long contexts in QA Pipeline?
For example, I use Pipeline from Huggingface Transformers to use a QA model card like this.
...
2
votes
2
answers
94
views
How to improve language model ex: BERT on unseen text in training?
I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure.
However, when I remove a ...
- 123
0
votes
1
answer
221
views
Adding a new token to a transformer model without breaking tokenization of subwords
I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
- 101
0
votes
0
answers
33
views
Code sanity check with one batch overfitting. Good value for loss?
In my current setup, I am trying to train a simple Bert model (DistilBert) for a classification task with 30 classes.
As is common, I performing a quick sanity check whether my code actually properly ...
1
vote
0
answers
79
views
How to Fine Tune a BERT model for sentiment analysis to get the best f1 score
I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score.
More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
- 199
0
votes
0
answers
32
views
Chatbot NLP model for inference on 11 GB GPU?
I’ve just found the amazing Huggingface library. It is an awesome piece of work.
I would like to train a chatbot on some existing dataset or several datasets (e.g. the Pile). For training (or fine-...
- 234
1
vote
1
answer
33
views
HuggingFace hate detection model
I am trying to train and evaluate a hate detection model using the HuggingFace Transformers library and this dataset. Model performance is secondary, just trying to get it going. I have preprocessed ...
- 13
0
votes
1
answer
52
views
How does T5 model work on input and target data while transfer learning?
I am working on a project where I want the model to generate job description based on Role, Industry, Skills. I have trained my data and got the resultant output.
I ...
- 33
0
votes
0
answers
472
views
Huggingface - TypeError: 'TensorSliceDataset' object is not subscriptable
I'm trying to make my own model for translate a language to another with T5ForConditionalGeneration and Huggingface using no pretrained model (I need to use my own dataset and tokenizer because no ...
- 101
1
vote
0
answers
18
views
Can a reformer model really handle long-range dependency?
I read this article about new attention model called Reformer. Here is the main strength of this model:
The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
0
votes
0
answers
27
views
How to generate sentences based on words?
I have a dataframe which has columns Role Name, Technical Skills, Soft Skills and average experience. I have to use these words ...
- 33
0
votes
0
answers
16
views
Multi TextClassification using DistillBERT
I am building a text classifier using DistillBERT from Huggingface which classify input text into Geography, Environment and Science , How could I increase the accuracy for first two labels as both ...
- 1
1
vote
0
answers
106
views
Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?
I'm doing sentiment analysis of Spanish tweets.
After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
- 131
1
vote
1
answer
209
views
How to use is_split_into_words with Huggingface NER pipeline
I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train.
My incoming text has already been split into words. When tokenizing during ...
- 111
0
votes
0
answers
30
views
Which steps are involved in sentiment analysis with Huggingface Transformers?
I want to perform a sentiment analysis of a dataset of (Spanish) tweets about COVID-19 vaccines.
I've already scraped the tweets and identified a pretrained model I can use for Spanish.
What I don't ...
- 131
0
votes
0
answers
12
views
Optimal method(s) to monitor attention matrices when doing training/inference using BERT-type models from transformers
Our team is using BERT/Roberta from the huggingface transformers library for sequence-classification (amongst other tasks). We are looking for an efficient way to monitor the attention matrices so as ...
- 199
0
votes
0
answers
65
views
Training Loss increases, but Validation Loss decreases
I am finetuning a T5 transformer model on a sequence to sequence task. My program outputs the training and validation loss every 500 optimization steps.
However, when I first started training the ...
- 43
4
votes
1
answer
90
views
How to measure the accuracy of an NLP paraphrasing model?
I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
- 43
0
votes
0
answers
6
views
How do pretrained models using SQUAD dataset work on an any other dataset?
I see in some Kaggle contests people have used models pretrained in SQUAD dataset for building QA systems for the dataset given in the contest. How does this work? How can a pretrained model in a ...
- 99
1
vote
0
answers
21
views
Personal Project Classifying Bank Account Data - NLP Noob [closed]
Background:
I'm looking to get up to speed with some of the newer ML classification techniques and keep my ML python fresh in my spare time/learn some new skills, so as a challenge I'm trying to see ...
- 11
1
vote
1
answer
568
views
HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00
I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
- 11
0
votes
0
answers
65
views
Are there any ways to check default values of pre-trained models before fine-tuning?
Background
According to the instruction on Hugging Face page, I'm trying to fine tune pre-trained model for named entity recognition.
I think I should try Transfer Learning for the first, but there is ...
- 129
0
votes
2
answers
102
views
Question answering bot: EM>F1, does it make sense?
I am fine-tuning a Question Answering bot starting from a pre-trained model from HuggingFace repo.
The dataset I am using for the fine-tuning has a lot of empty answers. So, after the fine tuning, ...
- 53
0
votes
0
answers
126
views
Pytorch DataLoader returns iterable , how and when to convert into a Tensor for Model Training
I have coded up two DataSet classes (one map-style and one iterator-style) to be used with DataLoader (either one is okay, but I just wanted to experiment and learn myself) on TEXT data.
The data + ...
- 101
1
vote
0
answers
105
views
How to do batch inference on Hugging face pretrained models?
I want to do batch inference on MarianMT model. Here's the code:
...
0
votes
0
answers
68
views
Does NSP task corrupt context during pre-training?
During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
- 101
1
vote
0
answers
18
views
Train and validation sets splits using load_data
I'm using the package "datasets". The code I have:
...
- 149
1
vote
0
answers
13
views
Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling
I intend to use 2 tasks of modelling including (a) Causal language modelling &
(b) Mask language modelling for training my new added tokens
My pseudo-code is below
...
- 11
1
vote
1
answer
112
views
Masked Language Modeling on Domain-specific Data
My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
4
votes
1
answer
1k
views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
- 297
1
vote
1
answer
148
views
dealing with HuggingFace's model's tokens
I have a few questions regarding tokenizing word/characters/emojis for different huggingface models.
From my understanding, a model would only perform best during inference if the token of the input ...
- 11
1
vote
1
answer
1k
views
How to do NER predictions with Huggingface BERT transformer
I am trying to do a prediction on a test data set without any labels for an NER problem.
Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
1
vote
0
answers
39
views
Top-K vs AUC - communicating results and next steps [closed]
I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of ...
- 111
2
votes
1
answer
521
views
How to i get word embeddings for out of vocabulary words using a transformer model?
When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
- 121
0
votes
0
answers
49
views
List of Google T5 possible operations
I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
- 101
0
votes
1
answer
101
views
which script can be used to finetune BERT fro squad question answering in hugging face library
I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used ...
- 113
0
votes
0
answers
112
views
BERT for classification model degenerates into all-positive predictions
As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
- 101
1
vote
0
answers
143
views
Huggingface Library - Multi-document summarization
Can BART, PEGASUS ... etc. API in huggingface library be used to directly perform multi document summarization?
(e.g. here: https://huggingface.co/transformers/model_doc/bart.html)
- 133