Tag Info

GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token. BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. "a", "W"). The subword vocabulary used by GPT-...

This article on Medium introduces GPT-3 makes some comparisons with BERT. Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still holds the edge. It’ s trained-on challenges which are better able to capture the latent relationship between text in different problem contexts." Also, in ...

BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text. In the case of GPT-2, OpenAI trained it only with <|endoftext|&...

What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that. However, you should take into account that it is possible that your model generates very similar sentences ...

BERT is a Transformer encoder, while GPT is a Transformer decoder: You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block. There is, however, an extra difference in how BERT and GPT are trained: BERT is a Transformer ...

I think there are (at least) two parts to take into account in evaluating such a model: Whether the generated text correctly relate to the input topic Whether the generated text is grammatically and semantically acceptable In my opinion the first kind of evaluation could reasonably be done with an automatic method such as the one you propose. Note that ...

BERT needs to be fine-tuned to do what you want. GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive) If you have enough data for fine-tuning, then per unit of compute (i.e. inference cost), you'll probably get much better performance out of BERT.

They are meant for different purposes and they are hardly comparable. RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it on your (potentially small) classification or tagging dataset. Some examples of tasks where RoBERTa is useful are sentiment classification, part-of-speech (POS) ...

Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.