Tag Info

Accepted

• 446
Accepted

How pre-trained BERT model generates word embeddings for out of vocabulary words?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
• 16k

How to get sentence embedding using BERT?

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the ...
• 2,349

What is purpose of the [CLS] token and why is its encoding output important?

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model ...
• 181
Accepted

What is a 'hidden state' in BERT output?

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer ...
• 859

Similarity of words using BERTMODEL

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on ...
• 859
Accepted

Implementation of BERT using Tensorflow vs PyTorch

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/...
• 16k
Accepted

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Here are the answers: In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
Accepted

Why is word prediction an obsession in Natural Language Processing?

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural ...
• 328

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

I think there are a few misconceptions in your statements. Please take into account the following BERT does not provide word-level representation. It provides sub-words embeddings and sentence ...
• 16k

Calculating cosine similarity between 3D arrays using Python

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the ...
• 14k

What is purpose of the [CLS] token and why is its encoding output important?

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, ...
• 61
Accepted

Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
• 16k

How can I add custom numerical features for training to BERT fine tuning?

With BERT I am assuming you are using finally the embeddings for your task. Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model ...
• 173
Accepted

What are the elements in a BERT word embedding?

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which ...
• 6,332
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
• 1,406

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Since many of your questions were answered already, I may only share my personal experience with your last question: 7) Is it a good idea to use BERT embeddings to get features for documents that can ...
• 101