Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

96 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
4
votes
0answers
152 views

Why do BERT classification do worse with longer sequence length?

I've been experimenting using transformer networks like BERT for some simple classification tasks. My tasks are binary assignment, the datasets are relatively balanced, and the corpus are abstracts ...
3
votes
1answer
65 views

Named Entity Recognition with BIO Tagging

I'm trying to implement NER using BIO annotation. For example "I went to the United States" [O, O, O, B, I, I] where B and I denote the beginning and '...
3
votes
0answers
51 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
3
votes
1answer
136 views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
3
votes
0answers
55 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
3
votes
2answers
879 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
3
votes
1answer
738 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
2
votes
0answers
30 views

Imbalance classes in Named Entity Recognition

I am currently working on a NER problem which attempts to extract 2 entities - place-of-interest(POI) and street from an address string in the Indonesian language. I used IndoBert (available here) and ...
2
votes
0answers
36 views

Social media text analysis

I'm currently analyzing Korean social media text. The below are the steps of the analysis. Collect/crawling text data from social media (e.g. Twitter, Facebook), which are related to specific topics. ...
2
votes
0answers
127 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
2
votes
3answers
897 views

Bert-Transformer : Why Bert transformer uses [CLS] token for classification instead of average over all tokens?

I am doing experiments on bert architecture and found out that most of the fine-tuning task takes the final hidden layer as text representation and later they pass it to other models for the further ...
2
votes
0answers
131 views

Remove subwords from BERT output

I'm trying to build a multilingual WSD system with BERT on top as the embedding layer. In order to have better performances, after BERT finishes its job (and performs Transfer Learning), I need to ...
2
votes
0answers
317 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
2
votes
0answers
86 views

Can BERT embeddings be used to reproduce the original content of the text?

From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different ...
2
votes
0answers
198 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
2
votes
1answer
512 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
2
votes
0answers
169 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , ...
1
vote
0answers
17 views

Nested Classification in Tensorflow

I am working on a text classification problem that has an output structure that one could consider nested in the sense that similar child labels fall under the umbrella of parent labels, which are ...
1
vote
0answers
9 views

What is the Intermediate (Dense) layer in between attention and output layers within an encoder block of a transformer in pytorch

Transformer (BERT) models in PyTorch have an intermediate dense layer in between attention and output layers whereas the BERT and Transformer papers just mention the attention connected directly to ...
1
vote
0answers
18 views

NER prections with distilbert transformer model

I am trying to extract 'agreement date' label from a corpus of legal contracts. In the train dataset, I used pytorch-transformer model to train. ...
1
vote
0answers
25 views

Is it possible to fine-tune a (Spanish RoBERTa) model for a different task?

I'm doing sentiment analysis of Spanish tweets. After reviewing some of the recent literature, I've seen that there's been a most recent effort to train a RoBERTa model exclusively on Spanish text. It ...
1
vote
0answers
17 views

How to get sentiment score for a word in a given dataset

I have a sentiment analysis dataset that is labeled in three categories: positive, negative, and neutral. I also have a list of words (mostly nouns), for which I want to calculate the sentiment value, ...
1
vote
0answers
26 views

How to do Bert Finetuning of failure cases?

I have a large dataset(public available) of text that is labelled. However the test distribution (actual production setting of company) while similar is not from the same source and thus tends to fail ...
1
vote
1answer
33 views

Document ranking on a web scraped dataset without any labelled data

I want to create a document ranking model which returns similar rows in the dataset for a sample query. The text in this corpus is standard english but without any labels (ie no query-related ...
1
vote
0answers
23 views

NLP-Problem, language model BERT?

Right now I am in the process of deciding on my masters thesis topic. Right now I and my professor are thinking about the possibility of having a large dataset of product requirements given in a ...
1
vote
1answer
131 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
1
vote
1answer
58 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...
1
vote
0answers
20 views

Extracting layer output from Classification model of SimpleTransformer

I have fine tuned a bert base model for text classification task. Now, I want to extract hidden layer output so as to combine this output with other features to train a random forest model. Problem ...
1
vote
1answer
243 views

Bert and SVM classification

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
1
vote
0answers
164 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
1
vote
0answers
133 views

NLP Bert model to to calculate text similarity, same sentence but not close similarity

Dear expert here: I have a simple program to calculate text similarity. The program is copied from internet. Initially, I have a list of sentences or stored in db and fetched from db, then I make the ...
1
vote
0answers
83 views

How to apply pruning on a BERT model?

I have trained a BERT model using ktrain (tensorflow wrapper) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production ...
1
vote
1answer
909 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
0answers
185 views

Using BERT for input embeddings in a seq2seq model

I'm currently trying to implement a paper that describes using BERT to embed inputs into a seq2seq model. "For word vectors, we use the deep contextualized word vectors from ELMo (Peters et al., ...
1
vote
0answers
16 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
1
vote
0answers
467 views

System Requirement to train BERT model

How much Hardware is required to train it well?(My current PC specs: 8GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU)) I have a dataset of approx 1lakh records.Is it is necessary to ...
1
vote
0answers
363 views

Training PCA on BERT word embedding: entire training dataset or each document?

I want to reduce the dimensionality of the BERT word embedding to, let's say, 50 dimensions. I am trying with PCA. I will use that for the document classification task. Now for training PCA, should ...
1
vote
0answers
306 views

Flair Custom NER

I'm working on a problem in the domain of NER. I have a dataset wherein I need to have custom tags for different entities. I don't know how to start or even where to start. I know that there are ...
1
vote
0answers
24 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
1
vote
0answers
143 views

fine tune BERT in a small GPU

I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each. My question: is it even possible ...
1
vote
0answers
215 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
0
votes
0answers
43 views

How should I set the threshold when I use a pre-trained natural-language-inference model as zero-shot text classifier?

I want to use a pre-trained natural-language-inference model as a zero-shot text classifier, using the transformers package. I have a set of topics, 'technology', 'science', 'politics', 'religion'. ...
0
votes
0answers
30 views

Using Sentence-Bert with other features in scikit-learn

I have a dataset where one feature is text and 4 more features. Sentence-Bert vectorizer transforms text data into tensors. I can use these sparse matrices directly with a machine learning classifier. ...
0
votes
0answers
12 views

BERT Text Classification Model gives error

I am fine-tuning a BERT Model for text classification with Tensorflow. Here is my code for building the model: ...
0
votes
0answers
11 views

How to improve accuracy? BERT

Dataframe: ...
0
votes
1answer
23 views

Is it possible to fine-tuning BERT by training it on multiple datasets? (Each dataset having it's own purpose)

BERT can be fine-tuned on a dataset for a specific task. Is it possible to fine-tune it on all these datasets for different tasks and then be utilized for these tasks instead of fine-tuning a BERT ...
0
votes
0answers
11 views

Multi TextClassification using DistillBERT

I am building a text classifier using DistillBERT from Huggingface which classify input text into Geography, Environment and Science , How could I increase the accuracy for first two labels as both ...
0
votes
0answers
13 views

How long does it take to fine-tune XLNet?

XLNet takes a lot more time than BERT during pre-training. This results in XLNet performing better than BERT in over 20 NLP tasks. How long does XLNet take for fine-tuning (let's assume this is ...
0
votes
0answers
11 views

Classify short sentences using BERT pre trained model with a custom dataset

I have a dataset that has 2 columns, an input and the class it is associated with. I have 6 classes and I am not able to find a way to train the BERT model on my dataset. I tried huggingface but I ...