Podcast #128: We chat with Kent C Dodds about why he loves React and discuss what life was like in the dark days before Git. Listen now.

Questions tagged [bert]

BERT stands for Bidirectional Encoder Representations from Transformers and is designed to pre-train deep bidirectional representations by jointly conditioning on both left and right context in all layers

Filter by
Sorted by
Tagged with
0
votes
1answer
11 views

How to obtain the word vectors optimally

I have a list of strings as shown sent_list = ["Carrefour is in France", "Apple pie is delicious", "Amazon has just delivered", ...] My code to get word ...
-1
votes
0answers
5 views

BERT pretraining hardware information [on hold]

I want to pre train BERT with a dataset of legal documents. Can I do it on google colab with TPU runtime ?
1
vote
2answers
18 views

Similarity of words using BERTMODEL

I want to find the similarity of words using the BERT model within the NER task. I have my own dataset so, I don't want to use the pre-trained model. I do the following: ...
0
votes
0answers
12 views

BERT for non-textual sequence data

I'm working on a deep learning solution for classifying sequence data that isn't raw text but rather entities (which have already been extracted from the text). I am currently using word2vec-style ...
1
vote
1answer
12 views

BERT Model Evaluation Measure in terms of Syntax Correctness and Semantic Coherence

For example I have an original sentence. The word barking corresponds to the word that is missing. ...
0
votes
0answers
19 views

Preprocessing for Text Classification in Transformer Models (BERT variants)

This might be silly to ask, but I am wondering if one should carry out the conventional text preprocessing steps for training one of the transformer models? I remember for training a W2V or Glove, ...
0
votes
1answer
37 views

BERT - How Question answering is different than classification

Basically I am trying to understand how question answering works in case of BERT. Code for both classes QuestionAnswering and Classification is pasted below for reference. My understanding is: <...
0
votes
1answer
17 views

TLDR Bot - Sentence Tagging w/ BERT

Currently making a bot that condenses news articles. I'm tagging sentences as important or not important using a simple BERT classifier. The results were... not great. I'm really interested in how I ...
0
votes
0answers
14 views

Best structure for a LSTM Bert sentence classifier

I'm interested in classifying sentences using BERT. Finetuning on a single sentence had very poor results. I'd like to add a forward and backward LSTM layer to try to improve results. I'm having ...
0
votes
1answer
78 views

How to get sentence embedding using BERT?

How to get sentence embedding using BERT? ...
0
votes
0answers
14 views

What will happen if we replace the transformer of BERT to evolved transformer?

If we replace the official BERT's transformer to evolved transformer, do the change accelerate the inference speed without losing accuracy?
1
vote
0answers
12 views

Can BERT embeddings be used to reproduce the original content of the text?

From what I understand, BERT provides contextualized embeddings that are not deterministic the way Word2Vec embeddings (i.e. the word "Queen" doesn't always produce the same vector, it'll be different ...
0
votes
0answers
13 views

What's the best way to store BERT training data (input IDs)

The tricky thing about the input IDs is what they're varying in length for each data sample, so regular hdf5 may not be ideal. Since Bert is so popular I am wondering if there's an established way to ...
7
votes
1answer
125 views

Why is word prediction an obsession in Natural Language Processing?

I have heard how great BERT is at masked word prediction, i.e. predicting a missing word from a sentence. In a Medium post about BERT, it says: The basic task of a language model is to predict ...
0
votes
1answer
106 views

BERT training on two tasks: what is the order of tasks?

I read that BERT has been trained on two tasks: Masked Language Modeling and Next Sentence Prediction. I want to gain clarity how exactly it was done. Was it initially trained on Masked Language ...
0
votes
1answer
41 views

Smallest Possible Dataset for Text Classification using BERT

What are your experiences for appropriate dataset sizes for usual text classification tasks using a finetuned BERT such as sentiment analysis? ~100 examples ~1000 examples ... ~10000000 examples ...
0
votes
0answers
35 views

Modifying BERT sentence encodings

I'm using BERT to encode sentences. The sentences I'm encoding are quite similar, meaning they all belong to the same overall topic. Therefor, I am using another parameter for measuring similarity. ...
2
votes
0answers
36 views

Structuring a LSTM Layer

I'm trying to improve an NER Bert sequence tagger using LSTM layers in TensorFlow. I'm a bit unclear on the interface and how a LSTM layer should be set up. Currently, I'm taking in 3-5 sentences and ...
1
vote
1answer
51 views

BERT : text classification and feature extractionn

I have tried multi-label text classification with BERT. Here is the sample input: $15.00 hour, customer service, open to industries One of the labels is Billing_rate and prediction score looks ...
0
votes
0answers
40 views

Summarization of documents using BERT

I have a collection of various documents that are partitioned according to their global topics. For each of these topics, I want to generate a new document that would summarize all of the ...
0
votes
0answers
342 views

BERT or ELMo for Document Similarity

Does anyone use BERT or ELMo language models to determine the similarity between two text documents? My question aims to collect all possible ways for combining the contextual word embeddings ...
2
votes
0answers
53 views

Predicting word from a set of words

My task is to predict relevant words based on a short description of an idea. for example "SQL is a domain-specific language used in programming and designed for managing data held in a relational ...
1
vote
1answer
258 views

How can I add custom numerical features for training to BERT fine tuning?

I have currently fine tuned the BERT model on some custom data and I want to conduct some more experiments to increase the accuracy. My original dataset consists of a pair of sentences (like MRPC ...
3
votes
0answers
322 views

How to add a CNN layer on top of BERT?

I am just playing with bert (Bidirectional Encoder Representation from Transformer) Research Paper Suppose I want to add any other model or layers like Convolutional Neural Network layers (CNN), Non ...
0
votes
0answers
29 views

Text classifiaction for large datasets using Transfer learning

I am trying to do text classification on a very large set of documents using the pretrained GPT model. The problem is GPT takes max sequence length $\le$ 1024. I can't truncate the data as I need to ...
0
votes
0answers
59 views

Finetuning BERT

Referring to the PyTorch port by huggingface of the native BERT library, I want to fine-tune the generated model on my personal dataset containing raw text. Could you please point out how this can be ...
1
vote
0answers
1k views

BERT vs Word2VEC: Is bert disambuguate the meaning of the word vector?

Word2vec: Word2vec provides a vector for each token/word and those vectors encode the meaning of the word. Although those vectors are not human interpretable, the meaning of the vectors are ...
1
vote
0answers
16 views

bert_large is inaccurate compared to bert_base

I have posted this question on github official site too - Issue 708 I have been using bert_Base for Question and answering. Mostly it is good, but for production grade it needs to answer more ...
2
votes
1answer
674 views

Calculating cosine similarity between 3D arrays using Python

I have two matrices with multiple columns and three rows each. I calculated the cosine similarity (sklearn) but it gives the result as a matrix. How can I obtain one single value? The matrices are the ...
3
votes
1answer
1k views

What is whole word masking in the recent BERT model?

I was checking BERT GitHub page and noticed that there are new models built from a new training technique called "whole word masking". Here is a snippet describing it: In the original pre-...
1
vote
1answer
112 views

Get long answers from BERT

We are using Google BERT for question and answering. We are using vanialla bert-base-uncased as well as squad trained checkpoints. The answers from BERT are very short and crisp. For example, if ...
2
votes
1answer
141 views

How to classify neutral sentiments using BERT

We can do text classification as positive and negative as mentioned in below notebook. But is there any way to classify neutral sentiment also? https://colab.research.google.com/github/google-...
0
votes
1answer
37 views

Incorrect Text Classification, But Accurate Model. Do I Perform Manual Text Classification For A Data Set?

I'm currently using Google's BERT pre-trained sentiment analysis model that is trained on an IMDb pos/neg review dataset. I'm using this model to predict whether tweets are positive (bullish) or ...
0
votes
0answers
42 views

Best service to host fine-tuned bert model?

I want to offer my fine-tuned bert model over the cloud. Are there any easy services to do this while only paying for usage (instead of an entire server)?
3
votes
0answers
687 views

BERT: it is possible to use it for topic modeling?

I'm struggling to understand which are the full capabilities of BERT: it is possible to make topic modeling of text, like the one we can achieve with LDA?
2
votes
1answer
374 views

Bert: fine-tuning the entire pre-trained model end-to-end vs using contextual token vector

In the official github page of BERT, it mentions that: In certain cases, rather than fine-tuning the entire pre-trained model end-to-end, it can be beneficial to obtained pre-trained contextual ...
2
votes
2answers
152 views

meaning of fine-tuning in nlp task

There are two types of transfer learning model. One is feature extraction, where the weights of the pre-trained model are not changed while training on the actual task and other is the weights of the ...
1
vote
1answer
871 views

what is the first input to the decoder in a transformer model?

The image is from url: Jay Alammar on transformers K_encdec and V_encdec are calculated in a matrix multiplication with the encoder outputs and sent to the encoder-decoder attention layer of each ...
2
votes
1answer
652 views

bert-as-service maximum sequence length

I installed bert-as-service (bert-as-service github repo) and tried encoding some sentences in Japanese on the multi_cased_L-12_H-768_A-12 model. It seems to work ...
0
votes
2answers
242 views

What is the use of [SEP] in paper BERT?

I know that [CLS] means the start of a sentence and [SEP] makes BERT know the second sentence has begun. However, I have a question. If I have 2 sentences, which are s1 and s2, and our fine-tuning ...
0
votes
1answer
170 views

Which is the “most properly working” Bert-Ner repository

I am trying to find a repository in Github to get a Pytorch-reimplementation of the Bert model for NER task. So far, I found the following repos: https://github.com/kamalkraj/BERT-NER https://github....
16
votes
2answers
5k views

What is GELU activation?

I was going through BERT paper which uses GELU (Gaussian Error Linear Unit) which states equation as $$ GELU(x) = xP(X ≤ x) = xΦ(x).$$ which appriximates to $$0.5x(1 + tanh[\sqrt{ 2/π}(x + 0.044715x^...
1
vote
1answer
110 views

BERT has a non deterministic behaviour

I am using the BERT implementation in https://github.com/google-research/bert for feature extracting and I have noticed a weird behaviour which I was not expecting: if I execute the program twice on ...
1
vote
0answers
128 views

Output range of BERT model shrinks after fine-tuning on domain specific dataset

My model's sigmoid output range has shrunk after transfer learning with small a dataset. My pretrained model has an output range of 0 to 1. After fine-tuning with a smaller domain specific dataset, ...
4
votes
0answers
3k views

How is WordPiece tokenization helpful to effectively deal with rare words problem in NLP?

I have seen that NLP models such as BERT utilize WordPiece for tokenization. In WordPiece, we split the tokens like playing to play and ##ing. It is mentioned that it covers a wider spectrum of Out-Of-...
2
votes
0answers
118 views

Paragraph Generator using BERT or GPT

I am trying to generate similar sentences, called paragraph generation. For example, what is the name of eldest brother of ram? - For this paragraphs can be - who is oldest brother of ram? , Who is ...
2
votes
1answer
665 views

Incrementally Train BERT with minimum QnA records - to get improved results

We are using Google BERT for Question and Answering. We have fine tuned BERT with SQUAD QnA release train data set (https://github.com/google-research/bert , https://rajpurkar.github.io/SQuAD-explorer/...
1
vote
0answers
384 views

How Transformer is Bidirectional - Machine Learning

Asking question in datascience forum, as this forum seems well suited for data science related questions: https://stackoverflow.com/questions/55158554/how-transformer-is-bidirectional-machine-learning/...
4
votes
1answer
1k views

Bert Fine Tuning with additional features

I want to use Bert for an nlp task. But I also have additional features that I would like to include. From what I have seen, with fine tuning, one only changes the labels and retrains the ...