Questions tagged [huggingface]
The huggingface tag has no usage guidance.
37
questions
1
vote
1answer
10 views
How does T5 model work on input and target data while transfer learning?
I am working on a project where I want the model to generate job description based on Role, Industry, Skills. I have trained my data and got the resultant output.
I ...
0
votes
0answers
4 views
Huggingface - TypeError: 'TensorSliceDataset' object is not subscriptable
I'm trying to make my own model for translate a language to another with T5ForConditionalGeneration and Huggingface using no pretrained model (I need to use my own dataset and tokenizer because no ...
1
vote
0answers
13 views
Can a reformer model really handle long-range dependency?
I read this article about new attention model called Reformer. Here is the main strength of this model:
The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
0
votes
0answers
21 views
How to generate sentences based on words?
I have a dataframe which has columns Role Name, Technical Skills, Soft Skills and average experience. I have to use these words ...
0
votes
0answers
10 views
Multi TextClassification using DistillBERT
I am building a text classifier using DistillBERT from Huggingface which classify input text into Geography, Environment and Science , How could I increase the accuracy for first two labels as both ...
1
vote
0answers
24 views
Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?
I'm doing sentiment analysis of Spanish tweets.
After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
0
votes
0answers
15 views
How to use is_split_into_words with Huggingface NER pipeline
I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train.
My incoming text has already been split into words. When tokenizing during ...
0
votes
0answers
18 views
Which steps are involved in sentiment analysis with Huggingface Transformers?
I want to perform a sentiment analysis of a dataset of (Spanish) tweets about COVID-19 vaccines.
I've already scraped the tweets and identified a pretrained model I can use for Spanish.
What I don't ...
0
votes
0answers
12 views
Optimal method(s) to monitor attention matrices when doing training/inference using BERT-type models from transformers
Our team is using BERT/Roberta from the huggingface transformers library for sequence-classification (amongst other tasks). We are looking for an efficient way to monitor the attention matrices so as ...
0
votes
0answers
28 views
Training Loss increases, but Validation Loss decreases
I am finetuning a T5 transformer model on a sequence to sequence task. My program outputs the training and validation loss every 500 optimization steps.
However, when I first started training the ...
2
votes
1answer
27 views
How to measure the accuracy of an NLP paraphrasing model?
I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
0
votes
0answers
5 views
How do pretrained models using SQUAD dataset work on an any other dataset?
I see in some Kaggle contests people have used models pretrained in SQUAD dataset for building QA systems for the dataset given in the contest. How does this work? How can a pretrained model in a ...
1
vote
0answers
17 views
Personal Project Classifying Bank Account Data - NLP Noob [closed]
Background:
I'm looking to get up to speed with some of the newer ML classification techniques and keep my ML python fresh in my spare time/learn some new skills, so as a challenge I'm trying to see ...
0
votes
1answer
123 views
HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00
I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
0
votes
0answers
29 views
Are there any ways to check default values of pre-trained models before fine-tuning?
Background
According to the instruction on Hugging Face page, I'm trying to fine tune pre-trained model for named entity recognition.
I think I should try Transfer Learning for the first, but there is ...
0
votes
2answers
74 views
Question answering bot: EM>F1, does it make sense?
I am fine-tuning a Question Answering bot starting from a pre-trained model from HuggingFace repo.
The dataset I am using for the fine-tuning has a lot of empty answers. So, after the fine tuning, ...
0
votes
0answers
21 views
Pytorch DataLoader returns iterable , how and when to convert into a Tensor for Model Training
I have coded up two DataSet classes (one map-style and one iterator-style) to be used with DataLoader (either one is okay, but I just wanted to experiment and learn myself) on TEXT data.
The data + ...
1
vote
0answers
66 views
How to do batch inference on Hugging face pretrained models?
I want to do batch inference on MarianMT model. Here's the code:
...
0
votes
0answers
33 views
Does NSP task corrupt context during pre-training?
During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
1
vote
0answers
12 views
Train and validation sets splits using load_data
I'm using the package "datasets". The code I have:
...
1
vote
0answers
12 views
Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling
I intend to use 2 tasks of modelling including (a) Causal language modelling &
(b) Mask language modelling for training my new added tokens
My pseudo-code is below
...
1
vote
1answer
57 views
Masked Language Modeling on Domain-specific Data
My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
3
votes
1answer
314 views
Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?
I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.).
For the purposes of this question, you may assume that I already have the input part ...
1
vote
1answer
76 views
dealing with HuggingFace's model's tokens
I have a few questions regarding tokenizing word/characters/emojis for different huggingface models.
From my understanding, a model would only perform best during inference if the token of the input ...
2
votes
1answer
683 views
How to do NER predictions with Huggingface BERT transformer
I am trying to do a prediction on a test data set without any labels for an NER problem.
Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
1
vote
0answers
25 views
Top-K vs AUC - communicating results and next steps [closed]
I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of ...
2
votes
1answer
286 views
How to i get word embeddings for out of vocabulary words using a transformer model?
When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
0
votes
0answers
36 views
List of Google T5 possible operations
I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
0
votes
1answer
76 views
which script can be used to finetune BERT fro squad question answering in hugging face library
I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used ...
0
votes
0answers
98 views
BERT for classification model degenerates into all-positive predictions
As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
0
votes
0answers
9 views
Does transformer learn from context to context
We are fine-tuning a transformer-based question answering model to answer the questions about the novels with the chronological plot. We are breaking up the novel into chapters and then using them as ...
1
vote
0answers
85 views
Huggingface Library - Multi-document summarization
Can BART, PEGASUS ... etc. API in huggingface library be used to directly perform multi document summarization?
(e.g. here: https://huggingface.co/transformers/model_doc/bart.html)
1
vote
0answers
83 views
How to apply pruning on a BERT model?
I have trained a BERT model using ktrain (tensorflow wrapper) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production ...
4
votes
1answer
447 views
Bert for QuestionAnswering input exceeds 512
I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
1
vote
1answer
896 views
Overfitting in Huggingface's TFBertForSequenceClassification
I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
1answer
81 views
Should weight distribution change more when fine-tuning transformers-based classifier?
I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation:
...
0
votes
1answer
582 views
BERT - How Question answering is different than classification
Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is:
<...
