Questions tagged [huggingface]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
4
votes
1answer
357 views

Bert for QuestionAnswering input exceeds 512

I'm training Bert on question answering (in Spanish) and i have a large context, only the context exceeds 512, the total question + context is 10k, i found that longformer is bert like for long ...
3
votes
1answer
90 views

Minimal working example or tutorial showing how to use Pytorch's nn.TransformerDecoder for batch text generation in training and inference modes?

I want to solve a sequence-to-sequence text generation task (e.g. question answering, language translation, etc.). For the purposes of this question, you may assume that I already have the input part ...
1
vote
1answer
36 views

Masked Language Modeling on Domain-specific Data

My goal is to have a language model that understands the relationships between words and can fill the masks in a sentence related to a specific domain. At first, I thought about pretraining or even ...
1
vote
0answers
10 views

Train and validation sets splits using load_data

I'm using the package "datasets". The code I have: ...
1
vote
0answers
10 views

Is it correct to load weights from task Masked Language Modeling to train Causal Language Modeling

I intend to use 2 tasks of modelling including (a) Causal language modelling & (b) Mask language modelling for training my new added tokens My pseudo-code is below ...
1
vote
1answer
44 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
1
vote
1answer
350 views

How to do NER predictions with Huggingface BERT transformer

I am trying to do a prediction on a test data set without any labels for an NER problem. Here is some background. I am doing named entity recognition using tensorflow and Keras. I am using huggingface ...
1
vote
0answers
23 views

Top-K vs AUC - communicating results and next steps [closed]

I have a bi-LSTM multi-label text classification model which when training on a highly imbalanced dataset with 1000 possible labels gives a top-k (k=5) categorical accuracy of 86% and a focal loss of ...
1
vote
1answer
162 views

How to i get word embeddings for out of vocabulary words using a transformer model?

When i tried to get word embeddings of a sentence using bio_clinical bert, for a sentence of 8 words i am getting 11 token ids(+start and end) because "embeddings" is an out of vocabulary ...
1
vote
0answers
74 views

How to apply pruning on a BERT model?

I have trained a BERT model using ktrain (tensorflow wrapper) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production ...
1
vote
1answer
666 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
1answer
59 views

Should weight distribution change more when fine-tuning transformers-based classifier?

I'm using pre-trained DistilBERT model from Huggingface with custom classification head, which is almost the same as in the reference implementation: ...
0
votes
1answer
57 views

which script can be used to finetune BERT fro squad question answering in hugging face library

I have gone through lot of blogs which talk about run_squad.py script from hugging face, but I could not find it in the latest repo. So, which script has to be used ...
0
votes
1answer
525 views

BERT - How Question answering is different than classification

Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is: <...
0
votes
0answers
18 views

Does NSP task corrupt context during pre-training?

During the pre-training of BERT, if we just use MLM our input will be: [CLS] SentenceA [SEP]. So if there is a masked token in Sentence A, it will be predicted by ...
0
votes
0answers
32 views

List of Google T5 possible operations

I am trying to use the huggingface.co pre-trained model of Google T5 (https://huggingface.co/t5-base) for a variety of tasks. But I can`t find a list of many tasks it really supports and how to ...
0
votes
0answers
86 views

BERT for classification model degenerates into all-positive predictions

As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
0
votes
0answers
7 views

Does transformer learn from context to context

We are fine-tuning a transformer-based question answering model to answer the questions about the novels with the chronological plot. We are breaking up the novel into chapters and then using them as ...
0
votes
0answers
52 views

Huggingface Library - Multi-document summarization

Can BART, PEGASUS ... etc. API in huggingface library be used to directly perform multi document summarization? (e.g. here: https://huggingface.co/transformers/model_doc/bart.html)
0
votes
0answers
74 views

PyTorch Hugging Face - Language generation with torchscript model

Have a fine-tuned a summarization model following the Hugging Face seq2seq guide (starting from sshleifer/distilbart-xsum-12-6). We are interested in using AWS elastic inference for deployment for ...
0
votes
0answers
110 views

huggingface bert - need help on working of code

Looking for some explanation of understanding of the BERT implementation by huggingface. I would explain my understanding below and then ask question: Below is code for question answering ...