Hot answers tagged

38

GELU function We can expand the cumulative distribution of $\mathcal{N}(0, 1)$, i.e. $\Phi(x)$, as follows: $$\text{GELU}(x):=x{\Bbb P}(X \le x)=x\Phi(x)=0.5x\left(1+\text{erf}\left(\frac{x}{\sqrt{2}}\right)\right)$$ Note that this is a definition, not an equation (or a relation). Authors have provided some justifications for this proposal, e.g. a ...


20

There is actually an academic paper for doing so. It is called S-BERT or Sentence-BERT. They also have a github repo which is easy to work with.


19

BERT can't be used for next word prediction, at least not with the current state of the research on masked language modeling. BERT is trained on a masked language modeling task and therefore you cannot "predict the next word". You can only mask a word and ask BERT to predict it given the rest of the sentence (both to the left and to the right of ...


16

CLS stands for classification and its there to represent sentence-level classification. In short in order to make pooling scheme of BERT work this tag was introduced. I suggest reading up on this blog where this is also covered in detail.


15

No, BERT is not a traditional language model. It is a model trained on a masked language model loss, and it cannot be used to compute the probability of a sentence like a normal LM. A normal LM takes an autoregressive factorization of the probability of the sentence: $p(s) = \prod_t P(w_t | w_{<t})$ On the other hand, BERT's masked LM loss focuses on ...


14

The need for an encoder depends on what your predictions are conditioned on, e.g.: In causal (traditional) language models (LMs), each token is predicted conditioning on the previous tokens. Given that the previous tokens are received by the decoder itself, you don't need an encoder. In Neural Machine Translation (NMT) models, each token of the translation ...


14

You are right that you actually do not need to know the architectures if you just want to apply them. But there are to reasons why it would be good to understand the architecture. Models often do not work off the shelf for your problem. In this case you will have to tune the model parameters etc. in order to apply the model to your problem. So knowledge of ...


13

[CLS] stands for classification. It is added at the beginning because the training tasks here is sentence classification. And because they need an input that can represent the meaning of the entire sentence, they introduce a new tag. They can’t take any other word from the input sequence, because the output of that is the word representation. So they add a ...


12

BERT and ELMo are recent advances in the field. However, there is a fine but major distinction between them and the typical task of word-sense disambiguation: word2vec (and similar algorithms including GloVe and FastText) are distinguished by providing knowledge about the constituents of the language. They provide semantic knowledge, typical about word types ...


11

At each decoding time step, the decoder receives 2 inputs: the encoder output: this is computed once and is fed to all layers of the decoder at each decoding time step as key ($K_{endec}$) and value ($V_{endec}$) for the encoder-decoder attention blocks. the target tokens decoded up to the current decoding step: for the first step, the matrix contains in ...


10

First note that $$\Phi(x) = \frac12 \mathrm{erfc}\left(-\frac{x}{\sqrt{2}}\right) = \frac12 \left(1 + \mathrm{erf}\left(\frac{x}{\sqrt2}\right)\right)$$ by parity of $\mathrm{erf}$. We need to show that $$\mathrm{erf}\left(\frac x {\sqrt2}\right) \approx \tanh\left(\sqrt{\frac2\pi} \left(x + a x^3\right)\right)$$ for $a \approx 0.044715$. For large values ...


9

Which vector represents the sentence embedding here? Is it hidden_reps or cls_head? If we look in the forward() method of the BERT model, we see the following lines explaining the return types: outputs = (sequence_output, pooled_output,) + encoder_outputs[1:] # add hidden_states and attentions if they are here return outputs # sequence_output, ...


8

BERT is a transformer. A transformer is made of several similar layers, stacked on top of each others. Each layer have an input and an output. So the output of the layer n-1 is the input of the layer n. The hidden state you mention is simply the output of each layer. You might want to quickly look into this explanation of the Transformer architecture : ...


7

First of all, I think you are confused with pretrained and finetuned. BERT is pretrained on a lot of text data. By using this pretrained BERT, you have a model that already have knowledge about text. BERT can then be finetuned on specific dataset, where BERT learn specific knowledge related to the dataset. That's why a finetuned BERT is bad on other ...


7

BERT does not provide word-level representations, but subword representations. You may want to combine the vectors of all subwords of the same word (e.g. by averaging them), but that is up to you, BERT only gives you the subword vectors. Subwords are used for representing both the input text and the output tokens. When an unseen word is presented to BERT, it ...


6

I think there are a few misconceptions in your statements. Please take into account the following BERT does not provide word-level representation. It provides sub-words embeddings and sentence representations. For some words, there may be a single subword while, for others, the word may be decomposed in multiple subwords. The representations of subwords ...


6

Your input matrices (with 3 rows and multiple columns) are saying that there are 3 samples, with multiple attributes. So the output you will get will be a 3x3 matrix, where each value is the similarity to one other sample (there are 3 x 3 = 9 such combinations) If you were to print out the pairwise similarities in sparse format, then it might look closer to ...


6

There are not only 2, but many implementations of BERT. Most are basically equivalent. The implementations that you mentioned are: The original code by Google, in Tensorflow. https://github.com/google-research/bert Implementation by Huggingface, in Pytorch and Tensorflow, that reproduces the same results as the original implementation and uses the same ...


6

The maximum input length is a limitation of the model by construction. That number defines the length of the positional embedding table, so you cannot provide a longer input, because it is not possible for the model to index the positional embedding for positions greater than the maximum. This limitation, nevertheless, is not arbitrary, but has a deeper ...


5

phil ##am #mon is a subword encoding of the single word “philammon” into 3 tokens. The comment just means that they mask words as opposed to tokens by taking into account subword encoding. For more on subword encodings take a look at the slides from cs224, especially Byte Pair Encoding, from the Feb 14 subwords lecture at http://web.stanford.edu/class/...


5

First let's understand why the format is like this. BERT was pretrained using the format [CLS] sen A [SEP] sen B [SEP]. It is necessary for the Next Sentence Prediction task : determining if sen B is a random sentence with no links with A or not. The [SEP] in the middle is here to help the model understand which token belong to which sentence. At ...


5

The first line of the BERT abstract is We introduce a new language representation model called BERT. The key phrase here is "language representation model". The purpose of BERT and other natural language processing models like Word2Vec is to provide a vector representation of words, so that the vectors can be used as input to neural networks for other ...


5

With BERT I am assuming you are using finally the embeddings for your task. Solution 1: Once you have embeddings, you can use them as features and with your other features and then build a new model for the task. Solution 2: Here you will play with the network. Now here left one is the normal BERT, in the right we have another MLP network to deal with ...


5

Some points first: BERT is a word embedding: BERT is both word and sentence embedding. It needs to be taken into account that BERT is taking the sequence of words in a sentence into account which gives you a richer embedding of words in a context but in classic embeddings (yes, after BERT we can call others "classic"!) you mostly deal with neighborhood i.e. ...


5

BERT cannot use GloVe embeddings, simply because it uses a different input segmentation. GloVe works with the traditional word-like tokens, whereas BERT segments its input into subword units called word-pieces. On one hand, it ensures there are no out-of-vocabulary tokens, on the other hand, totally unknown words get split into characters and BERT probably ...


5

Here are the answers: In sequence modeling, we expect a sentence to be ordered sequence, thus we cannot take random words (unlike bag of words, where we are just bothered about the words and not really the order). For example: In bag of words: "I ate ice-cream" and "ice-cream ate I" are same, while this is not true for the models that ...


4

You can import the pre-trained bert model by using the below lines of code: pip install pytorch_pretrained_bert from pytorch_pretrained_bert import BertTokenizer, BertModel, BertForNextSentencePrediction BERT_CLASS = BertForNextSentencePrediction # Make sure all the files are in same folder, i.e vocab , config and bin file PRE_TRAINED_MODEL_NAME_OR_PATH = ...


4

In order to better understand the role of [CLS] let's recall that BERT model has been trained on 2 main tasks: Masked language modeling: some random words are masked with [MASK] token, the model learns to predict those words during training. For that task we need the [MASK] token. Next sentence prediction: given 2 sentences, the model learns to predict if ...


4

Here're my understandings: (1)[CLS] appears at the very beginning of each sentence, it has a fixed embedding and a fix positional embedding, thus this token contains no information itself. (2)However, the output of [CLS] is inferred by all other words in this sentence, so [CLS] contains all information in other words. This makes [CLS] a good representation ...


4

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text. In the case of GPT-2, OpenAI trained it only with <|endoftext|&...


Only top voted, non community-wiki answers of a minimum length are eligible