Skip to main content

Questions tagged [huggingface]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1 vote
0 answers
742 views

Creating LLM chatbot using llama-index + langchain

As the title suggests: I'm trying to build a chatbot which his goal should be sort of like "chatgpt". The chatbot will be installed on Slack workspace and I'm struggling with which scope I ...
Ben's user avatar
  • 209
1 vote
0 answers
85 views

trainable weights in automodel and comparison with lora

if i use RobertaForSequenceClassification or AutoModelForSequenceClassification, are all the weights trained or only the new classification head trained also , i am also noticing for "roberta-...
prajwal rao's user avatar
0 votes
1 answer
261 views

Passing target text to gpt2 and T5 for fine tuning to learn text generation task

I have text with each line in following format: <text-1> some text-1 <text-2> some text-2 <text-3> some text-3 I want fine tune model to learn ...
RajS's user avatar
  • 105
1 vote
2 answers
1k views

Fine-tuned MLM based RoBERTa not improving performance

We have lots of domain-specific data (200M+ data points, each document having ~100 to ~500 words) and we wanted to have a domain-specific LM. We took some sample data points (2M+) & fine-tuned ...
Kalsi's user avatar
  • 11
2 votes
2 answers
3k views

Dynamic batching and padding batches for NLP in deep learning libraries

This is the usual way we train modern deep learning models for NLP, e.g. with Huggingface libraries where we have a fix length for the input no. of tokens/subwoords unit. https://huggingface.co/docs/...
alvas's user avatar
  • 2,430
0 votes
0 answers
1k views

RuntimeError: It looks like your LightningModule has parameters that were not used in producing the loss returned by training_step

I'm trying to use donut, which is a transformer model with a huggingface implementation, and pre-train it on a language it hasn't been yet on my desktop. Unfortunately the version of the stack ...
lte__'s user avatar
  • 1,350
1 vote
2 answers
87 views

Fine-tune GPT on sketch data (stroke-3)

These past days I have started a personal project where I would like to build a model that, given an uncompleted sketch, it can finish it. I was planning on using some pretrained models that are ...
ilved17's user avatar
  • 41
0 votes
0 answers
298 views

How to finetune a closed generative huggingface model?

I want to finetune a huggingface pretrained model on our internal documentation in a way it stats answering related questions. I could not find the adequate tutorial.
welu's user avatar
  • 131
0 votes
0 answers
912 views

How to deal with DataCollator and DataLoaders in Huggingface?

I have issues combining a DataLoader and DataCollator. The following code with DataCollatorWithPadding results in a ...
3r1c's user avatar
  • 101
0 votes
1 answer
72 views

error useing soft max gives outputs greater than 1

I am using Hugging Face AutoModelForSequenceClassification, model is roberta, using it for text classification. There are 3 classes. The output is: ...
prajwal rao's user avatar
0 votes
3 answers
977 views

Combining sentence embeddings of two different models (sBERT and mBERT)

I am working on a chatbot that helps students. So, I wanted to make use of bert model which has better performance on mathematics, which lead to me to math-bert, but the paper on it said that it was ...
Glinty's user avatar
  • 33
0 votes
1 answer
144 views

Possible NLP approaches to extract 'goals' from text

I am planning to take up an interesting NLP project. I want to extract 'goal' statements from lengthy reports. For example, the goals can be We would be reducing our carbon footprint by 50% by 2025 or ...
air cooled's user avatar
0 votes
1 answer
215 views

Extract the embedding from a specific layer of MarianModel

I am using using MarianModel from the hub of HuggingFace for a translation task. Now I want to extract the embedding from the output of the last MarianEncoderLayer ...
lenhhoxung's user avatar
1 vote
1 answer
81 views

What could cause pre-trained Opus-MT models have wildly varying inference time when being used with transformers library?

I have been testing pre-trained Opus-MT models ported to transformers library for python implementation. Specifically, I am using opus-mt-en-fr for English to French translation. And the tokenizer and ...
Shan Dou's user avatar
  • 131
1 vote
0 answers
379 views

Dataset Format for fine tuning deepset/roberta-base-squad2 hugging face transformer model

I have been trying to fine tune the roberta model for QnA to my specific domain (healthcare). I am unable to find the correct way to provide the dataset format to the tokenizer in order to fine tune ...
Tushar Sethi's user avatar
0 votes
2 answers
1k views

Creating class labels for custom DataSets efficiently (HuggingFace)

I have pandas dataframes - test & train,they both have text and label as columns as shown below - ...
relu's user avatar
  • 103
2 votes
1 answer
424 views

What Preprocessing is Needed for Semantic Search Using Pre-trained Hugging Face Transformers?

I am building a project for my bachelor thesis and am wondering how to prepare my raw data. The goal is to program some kind of semantic search for job postings. My data set consists of stored web ...
node_env's user avatar
0 votes
1 answer
52 views

Do I need training data in multiple languages for a multilingual transformer?

I am attempting to train a transformer which can categorize sentences into one of n categories. This model should be able to work with a number of different languages - English and Arabic in my case. ...
KOB's user avatar
  • 189
0 votes
1 answer
1k views

How to get all 3 labels' sentiment from finbert instead of the most likely label's?

I'm using bert to do sentiment analysis. I previous used cardiffnlp's twitter-roberta-base-sentiment, https://huggingface.co/cardiffnlp/twitter-roberta-base-sentiment. It gives the the usage on its ...
user900476's user avatar
0 votes
1 answer
298 views

What did Sentence-Bert return here?

I used sentence bert to embed sentences from this tutorial https://www.sbert.net/docs/pretrained_models.html ...
user900476's user avatar
1 vote
0 answers
193 views

Could Attention_mask in T5 be a float in [0,1]?

I was inspecting T5 model from hf https://huggingface.co/docs/transformers/model_doc/t5 . attention_mask is presented as ...
Dave's user avatar
  • 13
2 votes
0 answers
288 views

How to train a Task Specific Knowledge Distillation model using Hugging face model

I was referring to this code: https://github.com/philschmid/knowledge-distillation-transformers-pytorch-sagemaker/blob/master/knowledge-distillation.ipynb From @philschmid I could follow most of the ...
MAC's user avatar
  • 277
1 vote
0 answers
1k views

How to save hugging face fine tuned model using pytorch and distributed training

I am fine tuning masked language model from XLM Roberta large on google machine specs. When I copy the model using gsutil and subprocess from container to GCP ...
MAC's user avatar
  • 277
0 votes
0 answers
2k views

Hugging face Model Output 'last_hidden_state'

I am using the Huggingface BERTModel, The model gives Seq2SeqModelOutput as output. The output contains the past hidden states and the last hidden state. These are my questions What is the use of the ...
Fhunmie's user avatar
  • 17
0 votes
1 answer
26 views

Transformer similarity fine-tuned way too often predicts pairs as similar

I fine-tuned a transformer for classification to compute similarity between names. This is a toy example for the training data: ...
Simone's user avatar
  • 101
1 vote
1 answer
1k views

How to prepare texts to BERT/RoBERTa models?

I have an artificial corpus I've built (not a real language) where each document is composed of multiple sentences which again aren't really natural language sentences. I want to train a language ...
IsaacLevon's user avatar
0 votes
2 answers
2k views

Get sentence embeddings of transformer-based models

I want to get sentence embeddings of transformer-based models (Bert, Roberta, Albert, Electra...). I plan on doing mean pooling on the hidden states of the second last layer just as what bert-as-...
LGDGODV's user avatar
  • 145
2 votes
2 answers
277 views

How to improve language model ex: BERT on unseen text in training?

I am using pre-trained language model for binary classification. I fine-tune the model by training on data my downstream task. The results are good almost 98% F-measure. However, when I remove a ...
IS92's user avatar
  • 123
1 vote
2 answers
4k views

Adding a new token to a transformer model without breaking tokenization of subwords

I'm running an experiment investigating the internal structure of large pre-trained models (BERT and RoBERTa, to be specific). Part of this experiment involves fine-tuning the models on a made-up new ...
Jigsaw's user avatar
  • 121
1 vote
0 answers
373 views

How to Fine Tune a BERT model for sentiment analysis to get the best f1 score

I am building a multi-class sentiment analysis BERT model that's optimized to give the best f1 score. More specifically, I train each epoch by optimizing binary cross entropy per class, taking the ...
vgoklani's user avatar
  • 238
1 vote
1 answer
52 views

HuggingFace hate detection model

I am trying to train and evaluate a hate detection model using the HuggingFace Transformers library and this dataset. Model performance is secondary, just trying to get it going. I have preprocessed ...
Mughees Asif's user avatar
0 votes
1 answer
695 views

How does T5 model work on input and target data while transfer learning?

I am working on a project where I want the model to generate job description based on Role, Industry, Skills. I have trained my data and got the resultant output. I ...
10sha25's user avatar
  • 43
1 vote
0 answers
2k views

Huggingface - TypeError: 'TensorSliceDataset' object is not subscriptable

I'm trying to make my own model for translate a language to another with T5ForConditionalGeneration and Huggingface using no pretrained model (I need to use my own dataset and tokenizer because no ...
Tristan Bilot's user avatar
1 vote
0 answers
37 views

Can a reformer model really handle long-range dependency?

I read this article about new attention model called Reformer. Here is the main strength of this model: The Reformer pushes the limit of longe sequence modeling by its ability to process up to half a ...
Kenenbek Arzymatov's user avatar
1 vote
0 answers
370 views

Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?

I'm doing sentiment analysis of Spanish tweets. After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
LeLuc's user avatar
  • 131
4 votes
2 answers
2k views

How to use is_split_into_words with Huggingface NER pipeline

I am using Huggingface transformers for NER, following this excellent guide: https://huggingface.co/blog/how-to-train. My incoming text has already been split into words. When tokenizing during ...
Alan Buxton's user avatar
0 votes
0 answers
37 views

Which steps are involved in sentiment analysis with Huggingface Transformers?

I want to perform a sentiment analysis of a dataset of (Spanish) tweets about COVID-19 vaccines. I've already scraped the tweets and identified a pretrained model I can use for Spanish. What I don't ...
LeLuc's user avatar
  • 131
4 votes
1 answer
422 views

How to measure the accuracy of an NLP paraphrasing model?

I using the HuggingFace library to do sentence paraphrasing (given an input sentence, the model outputs a paraphrase). How am I supposed to compare the results of two separate models (one trained with ...
carrot_142's user avatar
1 vote
0 answers
28 views

Personal Project Classifying Bank Account Data - NLP Noob [closed]

Background: I'm looking to get up to speed with some of the newer ML classification techniques and keep my ML python fresh in my spare time/learn some new skills, so as a challenge I'm trying to see ...
user2065472's user avatar
2 votes
1 answer
3k views

HuggingFace Transformers is giving loss: nan - accuracy: 0.0000e+00

I am a HuggingFace Newbie and I am fine-tuning a BERT model (distilbert-base-cased) using the Transformers library but the training loss is not going down, instead ...
JasonExcel's user avatar
0 votes
2 answers
1k views

Question answering bot: EM>F1, does it make sense?

I am fine-tuning a Question Answering bot starting from a pre-trained model from HuggingFace repo. The dataset I am using for the fine-tuning has a lot of empty answers. So, after the fine tuning, ...
SilentCloud's user avatar
1 vote
0 answers
278 views

How to do batch inference on Hugging face pretrained models?

I want to do batch inference on MarianMT model. Here's the code: ...
sai_varshittha's user avatar
1 vote
0 answers
28 views

Train and validation sets splits using load_data

I'm using the package "datasets". The code I have: ...
Adolf Miszka's user avatar
1 vote
0 answers
16 views

Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling

I intend to use 2 tasks of modelling including (a) Causal language modelling & (b) Mask language modelling for training my new added tokens My pseudo-code is below ...
Lắc Lê's user avatar
1 vote
1 answer
460 views

Masked Language Modeling on Domain-specific Data

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
mitra mirshafiee's user avatar
7 votes
1 answer
10k views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
Pablo Messina's user avatar
1 vote
1 answer
287 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
user113789's user avatar
1 vote
1 answer
2k views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
Khachatur Mirijanyan's user avatar
1 vote
0 answers
118 views

Top-K vs AUC - communicating results and next steps [closed]

I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of ...
ML_Engine's user avatar
  • 111
3 votes
1 answer
1k views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
cerofrais's user avatar
  • 131