Hot answers tagged

39 votes
Accepted

What is GELU activation?

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}...
user avatar
  • 8,629
23 votes

How to get sentence embedding using BERT?

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.
user avatar
21 votes
Accepted

Can BERT do the next-word-predict task?

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you ...
user avatar
  • 16k
20 votes
Accepted

What is purpose of the [CLS] token and why is its encoding output important?

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog ...
user avatar
  • 5,321
16 votes

What is purpose of the [CLS] token and why is its encoding output important?

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire ...
user avatar
  • 261
15 votes
Accepted

Is BERT a language model?

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes ...
user avatar
  • 16k
14 votes
Accepted

Why is the decoder not a part of BERT architecture?

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given ...
user avatar
  • 16k
14 votes

Why should I understand AI architectures?

You are right that you actually do not need to know the architectures if you just want to apply them. But there are to reasons why it would be good to understand the architecture. Models often do not ...
user avatar
13 votes

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms ...
user avatar
11 votes
Accepted

what is the first input to the decoder in a transformer model?

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value (...
user avatar
  • 16k
10 votes
Accepted

What is whole word masking in the recent BERT model?

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding. For more ...
user avatar
10 votes

What is the use of [SEP] in paper BERT?

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence ...
user avatar
  • 859
10 votes

What is GELU activation?

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show ...
user avatar
10 votes
Accepted

How pre-trained BERT model generates word embeddings for out of vocabulary words?

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, ...
user avatar
  • 16k
9 votes

How to get sentence embedding using BERT?

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the ...
user avatar
  • 2,349
8 votes

What is purpose of the [CLS] token and why is its encoding output important?

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model ...
user avatar
8 votes
Accepted

What is a 'hidden state' in BERT output?

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer ...
user avatar
  • 859
7 votes

Similarity of words using BERTMODEL

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on ...
user avatar
  • 859
7 votes
Accepted

Implementation of BERT using Tensorflow vs PyTorch

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/...
user avatar
  • 16k
7 votes
Accepted

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Here are the answers: In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not ...
user avatar
6 votes
Accepted

Why is word prediction an obsession in Natural Language Processing?

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural ...
user avatar
6 votes

BERT vs Word2VEC: Is bert disambiguating the meaning of the word vector?

I think there are a few misconceptions in your statements. Please take into account the following BERT does not provide word-level representation. It provides sub-words embeddings and sentence ...
user avatar
  • 16k
6 votes

Calculating cosine similarity between 3D arrays using Python

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the ...
user avatar
  • 14k
6 votes

What is purpose of the [CLS] token and why is its encoding output important?

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, ...
user avatar
6 votes
Accepted

Bert for QuestionAnswering input exceeds 512

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not ...
user avatar
  • 16k
5 votes

How can I add custom numerical features for training to BERT fine tuning?

With BERT I am assuming you are using finally the embeddings for your task. Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model ...
user avatar
5 votes
Accepted

What are the elements in a BERT word embedding?

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which ...
user avatar
5 votes
Accepted

Does BERT use GLoVE?

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called ...
user avatar
  • 1,406
5 votes

How should I use BERT embeddings for clustering (as opposed to fine-tuning BERT model for a supervised task)

Since many of your questions were answered already, I may only share my personal experience with your last question: 7) Is it a good idea to use BERT embeddings to get features for documents that can ...
user avatar
  • 101
5 votes
Accepted

What is the difference between BERT architecture and vanilla Transformer architecture

The name provides a clue. BERT (Bidirectional Encoder Representations from Transformers): So basically BERT = Transformer Minus the Decoder BERT ends with the final representation of the words after ...
user avatar
  • 673

Only top scored, non community-wiki answers of a minimum length are eligible