Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

68 questions with no upvoted or accepted answers
Filter by
Sorted by
Tagged with
5
votes
0answers
9k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
3
votes
1answer
34 views

Detecting grammatical errors with BERT

We fine-tuned BERT (bert-base-uncased) model with CoLA dataset for sentence classification task. The dataset is a mix of ...
3
votes
0answers
53 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
3
votes
2answers
815 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
3
votes
1answer
658 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
2
votes
0answers
16 views

Imbalance classes in Named Entity Recognition

I am currently working on a NER problem which attempts to extract 2 entities - place-of-interest(POI) and street from an address string in the Indonesian language. I used IndoBert (available here) and ...
2
votes
0answers
34 views

Social media text analysis

I'm currently analyzing Korean social media text. The below are the steps of the analysis. Collect/crawling text data from social media (e.g. Twitter, Facebook), which are related to specific topics. ...
2
votes
0answers
24 views

Medical NER for French language

I'm currently exploring the options to extract medical NER specifically for French language. I tried SpaCy's general French NER but it wasn't helpful to the cause (...
2
votes
0answers
70 views

Loss first decreases and then increases

I am using pre-trained xlnet-base-cased model and training it further on real vs fake news detection dataset. I noticed a trend in accuracy for first epoch. ...
2
votes
0answers
95 views

Remove subwords from BERT output

I'm trying to build a multilingual WSD system with BERT on top as the embedding layer. In order to have better performances, after BERT finishes its job (and performs Transfer Learning), I need to ...
2
votes
2answers
48 views

User profiling based on multiple posts

I currently have collected a dataset of different social media posts for each user with labels assigned to each user. I tried to use LSTM, and BERT for the text classification problem, So for each ...
2
votes
0answers
174 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
2
votes
1answer
420 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
2
votes
0answers
157 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of the eldest brother of ram? - For these paragraphs can be - who is the oldest brother of ram? , ...
1
vote
1answer
17 views

BERT is running out of memory in forward pass for my dictionary

Running code from this answer, my BERT is running out for my 4k words dictionary. I don't need to do anything with these words yet, just make embeddings for my data. So, using this exactly: ...
1
vote
1answer
33 views

dealing with HuggingFace's model's tokens

I have a few questions regarding tokenizing word/characters/emojis for different huggingface models. From my understanding, a model would only perform best during inference if the token of the input ...
1
vote
0answers
16 views

Extracting layer output from Classification model of SimpleTransformer

I have fine tuned a bert base model for text classification task. Now, I want to extract hidden layer output so as to combine this output with other features to train a random forest model. Problem ...
1
vote
1answer
81 views

Bert and SVM classification

I'm trying to understand the concepts in the title and how they fit into the task of binary classification. According to my understanding so far, you can encode text using various feature-extraction ...
1
vote
0answers
89 views

How do I handle class imbalance for text data when using pretrained models like BERT?

I have a skewed dataset consisting of samples of the form: Category 1 10000 Category 2 2000 Category 3 400 Category 4 300 Category 5 100 The dataset ...
1
vote
0answers
79 views

NLP Bert model to to calculate text similarity, same sentence but not close similarity

Dear expert here: I have a simple program to calculate text similarity. The program is copied from internet. Initially, I have a list of sentences or stored in db and fetched from db, then I make the ...
1
vote
0answers
72 views

How to apply pruning on a BERT model?

I have trained a BERT model using ktrain (tensorflow wrapper) to recognize emotion on text, it works but it suffers from really slow inference. That makes my model not suitable for a production ...
1
vote
1answer
550 views

Overfitting in Huggingface's TFBertForSequenceClassification

I'm using Huggingface's TFBertForSequenceClassification for multilabel tweets classification. During training the model archives good accuracy, but the validation accuracy is poor. I've tried to solve ...
1
vote
0answers
149 views

Using BERT for input embeddings in a seq2seq model

I'm currently trying to implement a paper that describes using BERT to embed inputs into a seq2seq model. "For word vectors, we use the deep contextualized word vectors from ELMo (Peters et al., ...
1
vote
0answers
13 views

How to convert subword PPL to word level PPL?

I'm using this formula to covert subword perpexity to word perplexity: PPL_word = exp(log(PPL_subword) * num_subwords / num_words) The question is do I need to ...
1
vote
0answers
386 views

System Requirement to train BERT model

How much Hardware is required to train it well?(My current PC specs: 8GB RAM, i5 2 core Processor, Standard GPU (No work going on GPU)) I have a dataset of approx 1lakh records.Is it is necessary to ...
1
vote
0answers
307 views

Training PCA on BERT word embedding: entire training dataset or each document?

I want to reduce the dimensionality of the BERT word embedding to, let's say, 50 dimensions. I am trying with PCA. I will use that for the document classification task. Now for training PCA, should ...
1
vote
0answers
224 views

Flair Custom NER

I'm working on a problem in the domain of NER. I have a dataset wherein I need to have custom tags for different entities. I don't know how to start or even where to start. I know that there are ...
1
vote
0answers
23 views

Multimodal end-to-end deep learning

I'm thinking of working on a project that involves multiple models of data and wanted to share my thoughts to get some feedback. Think of problem of sentiment classification where the input contains ...
1
vote
0answers
117 views

fine tune BERT in a small GPU

I want to to fine tune the BERT base model but the only accelerated HW I have access to is a couple of Quadro 600 GPUs which only pack 1GB RAM and 96 CUDA cores each. My question: is it even possible ...
1
vote
0answers
160 views

Weight matrices in transformers

I am trying to understand the transformer architecture. I am aware that the encoder/decoder contains multiple stacked self attention layers. Further each layer contains multiple heads. For example ...
1
vote
0answers
210 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
0answers
59 views

Can BERT embeddings be used to reproduce the original content of the text?

From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different ...
0
votes
1answer
27 views

Combining textual and numeric features into pre-trained Transformer BERT

I have a dataset with 3 columns: Text Meta-data (intending to extract features from it, then use those i.e., numerical features) Target label Question 1: How can I use a pre-trained BERT instance on ...
0
votes
1answer
17 views

Embedding from Transformer-based model from paragraph or documnet (like Doc2Vec)

I have a set of data that contains the different lengths of sequences. On average the sequence length is 600. The dataset is like this: ...
0
votes
0answers
41 views

Cluster images labels in some given categories using word embeddings

Given: set of images Labels in string format each one. Also I've given a set of Categories, also in string. ($Images \neq Categories $) Goal: I need to map given labels to given categories to "...
0
votes
0answers
12 views

NLP Style Transfer: Using my iMessage data to do predictive text generation in the style of my friends and family

So I wanted to use my iMessage data to train a model to see how the people I frequently text would fill in the blanks in sentences like "I am going to the [MASK]" or "I want to [MASK] ...
0
votes
1answer
18 views

Is LSTM or pretrained BERTForMasked LM usable for predicting changed word in a sentence using a small dataset? (2000 samples)

I have a small (2000 samples) dataset of newspaper headlines and their humorous conterparts where only one word is changed to sound silly, for example: Original headline: Police <officer> ...
0
votes
1answer
150 views

Where can I find documentation or paper mentioning pre-trained distilbert-base-nli-mean-tokens model?

I am trying to find more information about pre-trained model distilbert-base-nli-mean-tokens. Can someone please point me to it's paper or documentation? Is it ...
0
votes
0answers
28 views

How to train BERT (multi label) on imbalanced dataset for search query category classification

I have a dataset of 2 million search queries relative to 7000 categories. same query could have multiple categories. Aim is to predict category/categories for query with confidence score. I tried ...
0
votes
0answers
59 views

Use Categorical features in BERT model

I am trying to fine-tune BERT-base model for binary text classification using multiple features. 3 text features, 4 categorical features. Text features having more than 500 tokens length, and four ...
0
votes
0answers
12 views

Compact model for on-device next word prediction

I'm an iOS developer with no production ML experience besides some pet projects. The task is to create a model to predict the next word in English for a custom keyboard. I'd actually prefer to just ...
0
votes
0answers
24 views

Is it possible to use unlabeled text articles for summarization when fine tuning BERT?

I know that unlabeled data could be used in pre-training but if I want to do a fine tuning of unlabeled articles for summarization, is it mandatory that the articles are labeled with existing ...
0
votes
0answers
37 views

How to determine sentence similarity labels for sentence transformer fine-tuning?

I'm using the Sentence Transformer library to fine-tune pre-trained transformer models. In the fine tuning documentation, the example provided requires labels (from 0 to 1) that indicate the ...
0
votes
0answers
11 views

In Roberta if there is no NSP how are the weights update during training

in Roberta if NSP loss is not used , what is the loss function for DOC-SENTENCES or FULL-SENTENCES task , on what basis will it update the weights ?
0
votes
0answers
79 views

BERT for classification model degenerates into all-positive predictions

As a learning project, I'm training a BERT model with the CoLA dataset to detect sentence acceptability. Unfortunately my model is learning to classify every instance as "acceptable", and I'...
0
votes
0answers
7 views

Does transformer learn from context to context

We are fine-tuning a transformer-based question answering model to answer the questions about the novels with the chronological plot. We are breaking up the novel into chapters and then using them as ...
0
votes
0answers
13 views

Where does BERT fit in the Machine Learning Hierarchy?

I am a newbie in the machine learning world and I need guidance from the professionals. I am trying to make a hierarchy starting from machine learning, then to deep learning and to BERT. I have read ...
0
votes
0answers
62 views

Using bert (or fitbert) for predicting masked words from word candidates

Fitbert (which is based on Bert) can be used to predict (fill in) a masked word from a list of candidates as below: ...
0
votes
1answer
21 views

Using BERT for the first time, what are the two columns of my test_results.tsv?

I followed the steps to feed in both dev, test, train.tsv to the model, trained it, then tried to classify test data, and I only have 1 feature, and the classification is binary, 1 or 0. I assumed my ...
0
votes
0answers
50 views

Visualization of transformed features in BERT

So I'm trying the Intent Recognition with BERT using Keras and TensorFlow 2 available at kdnuggets.com and this is the code for the results evaluation. ...