6
GPT-2 does not use a word-level vocabulary but a subword-level vocabulary, specifically byte-pair encoding (BPE). This means that it does not predict the next word but the next subword token.
BPE tries to find the pieces of words that are most reusable. BPE also keeps character subwords (e.g. "a", "W").
The subword vocabulary used by GPT-...
4
This article on Medium introduces GPT-3 makes some comparisons with BERT.
Specifically, section 4 examines how GPT-3 and BERT differ and mentions that: "On the Architecture dimension, BERT still holds the edge. It’ s trained-on challenges which are better able to capture the latent relationship between text in different problem contexts."
Also, in ...
4
BERT is not trained with this kind of special tokens, so the tokenizer is not expecting them and therefore it splits them as any other piece of normal text, and they will probably harm the obtained representations if you keep them. You should remove these special tokens from the input text.
In the case of GPT-2, OpenAI trained it only with <|endoftext|&...
2
BERT needs to be fine-tuned to do what you want.
GPT-3 cannot be fine-tuned (even if you had access to the actual weights, fine-tuning it would be very expensive)
If you have enough data for fine-tuning, then per unit of compute (i.e. inference cost), you'll probably get much better performance out of BERT.
2
What you can do is to compare against a validation set of the same domain. First, you use your LM to generate many sentences, and, for each sentence, you compute the BLEU score against the whole validation set. This python script may be useful for that.
However, you should take into account that it is possible that your model generates very similar sentences ...
2
OpenAI has not released the weights of GPT-3, so you have to access it through their API. However, all other popular models have been released and are easily accessible. This includes GPT-2, BERT, RoBERTa, Electra, etc.
The easiest way to access them all in a unified way is by means of the Transformers Python library by Huggingface. This library supports ...
2
BERT is a Transformer encoder, while GPT is a Transformer decoder:
You are right in that, given that GPT is decoder-only, there are no encoder attention blocks, so the decoder is equivalent to the encoder, except for the masking in the multi-head attention block.
There is, however, an extra difference in how BERT and GPT are trained:
BERT is a Transformer ...
1
The parameters in GPT-3, like any neural network, are the weights and biases of the layers.
From the following table taken from the GTP-3 paper
there are different versions of GPT-3 of various sizes. The more layers a version has the more parameters it has since it has more weights and biases. Regardless of the model version, the words it was trained on are ...
1
This may be best understood with a bit more of context from the article:
A more fundamental limitation of the general approach described in this paper – scaling up any LM-like model, whether
autoregressive or bidirectional – is that it may eventually run into (or could already be running into) the limits of the
pretraining objective. Our current objective ...
1
The task is a specific case of NER (technically NER is a sequence labeling task, a special case of classification).
I think you would have two main options:
Apply a pre-trained NER model: most deal with time entities but not always very accurately, and it wouldn't be specifically adapted to your data so you wouldn't obtain the distinction between the three ...
1
GPT-2 is a causal language model. This means that, by default, it receives either no input at all or the initial tokens of a sentence/paragraph. It then completes whatever it was passed as input. Therefore, it is not meant to be used the way you are trying to do it.
Normally, in order to do conditional text generation, people use an encoder-decoder ...
1
I think there are (at least) two parts to take into account in evaluating such a model:
Whether the generated text correctly relate to the input topic
Whether the generated text is grammatically and semantically acceptable
In my opinion the first kind of evaluation could reasonably be done with an automatic method such as the one you propose. Note that ...
1
No - GPT-3 API is not currently public.
However once you get access, the documentation can be found at https://beta.openai.com/api-ref.
1
They are meant for different purposes and they are hardly comparable.
RoBERTa is meant for text classification and tagging tasks. The idea is that you take a pretrained RoBERTa model and finetune it on your (potentially small) classification or tagging dataset. Some examples of tasks where RoBERTa is useful are sentiment classification, part-of-speech (POS) ...
1
Yes fine-tuning GPT2 could help you through this objective. But the only concern is regarding the size of input data you have. To get a better performing model, you must a have larger input set.
1
My understanding is that transformer decoders and transformer encoder-decoder models typically operate in the way that the GPT-2 does, i.e., representations in the generated sequence are computed once and then reused for future steps. But you are correct that this is not the only way things can be done. One could recompute the representations for all tokens ...
1
The past token internal states are reused both in GPT-2 and any other Transformer decoder.
For example, in fairseq's implementation of the transformer, these previous states are received in TransformerDecoder.forward in parameter incremental_state(see the source code).
Remember that there is a mask in the self-attention blocks in the decoder that prevents ...
1
Limit outputs od decoder to N. Not sure how easy it would be, probably a bit digging into official implementation but after that the main "skeleton" of the GPT2 is usable, meaning that all of the pre-training can be reused to produce meaningful sentences.
Only top voted, non community-wiki answers of a minimum length are eligible